Item12479: preserve existing topic user mapping while migrating to ldap user mapping

The Case

Prior to implementing LdapContrib, if users have been able to choose their own WikiName trough registration after logging in trough SSO, how could these WikiNames be preserved? This is for example the case when migrating from TopicUserMapping to LdapContrib.

For example, a user with username "john" and full name "John Bert Doe" have selected the WikiName "JohnDoe" himself using the registration page.

After the implementation of LdapContrib, a likely scenario is that the LdapContrib will automatically map the username "john" to WikiName "JohnBertDoe", since this is the attributes returned from LDAP.

This is not a good case. - Firstly, username "john" will no longer have access to topics that included his old WikiName in for example "Set ALLOWTOPICVIEW". - Secondly, a different user, "john2" which actually is named "John Doe" (and not "John Bert Doe") would now get user "john"'s WikiName, and therefore all his access privileges.

Suggestion

1. Add {Ldap}{preserveWikiNames} = '/path/to/file.txt' to LocalSite.cfg. This files is essentially showing already existing relationships between usernames and WikiNames, which might not be the same as generated by LdapContrib.
2. In Contrib::LdapContrib, when refreshCache is called, if the cache is empty or $mode = 2 (nuke previous decisions), this list will be taken into consideration first, to ensure that the user still has his old WikiName after the implementation.
   • The preserveWikiNames file could for example contain one "loginname | WikiName" per line, and the rest is fetched from LDAP, based on the loginname.

Any feedback?

-- TerjeNessAndersen - 02 Jul 2013

In LdapContrib::refreshUsersCache, line 919 (in current version downloaded from /Extensions/LdapContrib), the cacheUserFromEntry sub is called. This accepts a explicit wikiname to be given. My idea is to generate a hash of already existing mappings from username to wikiname just before this, and then provide cacheUserFromEntry with the already existing wikiname, should it exist.

Like this: http://pastie.org/private/u1ehdfsb4ckg1xpb3dhomq

There seem to be one catch though. If a user mapping for "user1" exists in the file, he will surely get this wikiname, but only if no other users in LDAP maps to the same name (generated from {WikiNameAttributes}). If it is, someone can take the name before the search have reached "user1", then it will be a name clash, and who will then win? Some more research needed

-- TerjeNessAndersen - 25 Apr 2013

I'd like to have this feature. A further requirement I'd like is to map old login names to WikiName even when the login name is not in LDAP anymore - policy here is to nuke people out of LDAP when they leave the company. The problem with that is that old references to jlevens remain jlevens and not JulianLevens. The latter of course links and helps anybody understand the history of topics.

I suppose I should raise a separate task for this though

-- JulianLevens - 25 Apr 2013

I see.. Well, taking a quick look into LdapContrib, these fields are written to the db file:

• $data->{"U2W::$loginName"} = $wikiName;
• $data->{"W2U::$wikiName"} = $loginName;
• $data->{"DN2U::$dn"} = $loginName;
• $data->{"U2DN::$loginName"} = $dn;
• $data->{"U2EMAIL::$loginName"} = join(',', @$emails);

• $data->{WIKINAMES} = join(',', keys %wikiNames);
• $data->{LOGINNAMES} = join(',', keys %loginNames);

Which implications will leaving out "DN2U", "U2DN" and "U2EMAIL" have? There are most probably several consequences that need to be thought over.

---

As for the progress on this task, I have changed the logic a bit to deal with the problem i mentioned earlier today. It goes like this.

• In refreshUsersCache, if a {ExistingUsersFile} is set AND the existing cache is either A) empty or b) refreshldap is set to forced, a hash of existing user mappings is generated based on the given file (login => wikiname).
  • A $processExistingUsers flag is set to true
  • In the while loop polling ldap, the entries (one per user from ldap), is either put into @processFirst -if- this login is defined in the %existingUser, and in @processLast if it's not.
  • After the while loop, a new while loop occurs where @processFirst is processed first, and @processLast last. cacheUserFromEntry is called, which accepts a explicit wikiName to be set. For all entries in @processFirst, a wikiName exists, but for @processLast undef is sent so that the cacheUserFromEntry logic generates one.
  • When there is a collision, the existing users are already safe and sound in the cache, and the users from @processLast has to get a new WikiName, not taking presedence.

In other words, when fetching all the users from LDAP, the logins in {ExistingUsersFile} -must- be processed first, so that if any name clash should occur, it's not the user present in this file that "loses" his WikiName.

Any comments?

-- TerjeNessAndersen - 25 Apr 2013

The purpose of DN2U, U2DN, U2EMAIL:

• DN2U is required for looking up group members when indirection is active (Active Directory Server, for instance, stores DNs in a group's member attribute). It's also needed for resolving WikiName clashes, and for persisting the resolution.
• U2DN is required when authenticating wiki users (i.e. mainly with LdapTemplateLogin), to do the bind on the LDAP server.
• U2EMAIL is required for notify features (WebNotify, actionnotify, ...).

-- JanKrueger - 25 Apr 2013

A little update: I have begun with the implementation of this.

• A topic containing existing WikiName <-> login mappings can be set, if wanted.
  • This topic has to define these rules as " * WikiName - login" (3xspace-asterisk-space-WikiName-space-dash-space-login"
• This topic is taken into consideration when refresh is set to 'force', or if the cache is empty.
• The topic is checked, and rejected, if containing:
  • Duplicate WikiNames
  • Duplicate logins
  • WikiNames that do not evaluate true to Func::isValidWikiWord
  • logins that do not evaluate true to cfg::{'LoginNameFilterIn'}
• If the topic is checked out fine, all these rules will be processed first, ensuring that the existing login <-> WikiName mappings are preserved.

NB1: Also, concerning deleted users on LDAP server but not in the wiki: I will add support for a special setting cfg{Ldap}{GhostLoginName}. This value for login is allowed to be duplicate, meaning that the only duplicate is this, the existing login <-> WikiName mappings topic will be processed.

Any mappings from this value to a WikiName will lead to the WikiName to be "frozen", not being able to be taken by any login in the future. This to ensure that any existing access rights set to this WikiName will not be snatched by anyone else.

Julian: For unique login <-> WikiName mappings where the login isnt present on the LDAP server, I will try to preserve the mapping in the cache, meaning that only "U2W" and "W2U" will be set for this. I hope that this will not cause big problems, I guess i will find out!

NB2: I am also planning on releasing a web admin panel where admins can do the following actions:

• Edit a cached WikiName for a login.
• Delete a user from the cache
• Put a WikiName in the "do-not-touch" list, preventing users from grabbing that name.

...among other things

---

I am currently developing this for TWiki, but I will most likely modify it and release it here also, if wanted

- Terje

-- TerjeNessAndersen - 27 Jun 2013

Terje, did you take a look at the {RewriteWikiNames} feature? It lets you specify a hash of rules to post-process wiki names produced automatically.

In your use case as described above this would be

$Foswiki::cfg{Ldap}{RewriteWikiNames} = {
   # regex => result string
   'JohnBertDoe' => 'JohnDoe',

   # another use case: strip off domain names in cases where the wiki name is derived from mail
  '^(.*)@.*$' => '$1'
};

As such the feature you are looking for is already there, albeit stored in LocalSite.cfg and not in a wiki topic.

I am not a fan of having LDAP configuration parameters split up between configure and some wiki topic. I'd rather keep it all in one place so people can see all relevant configuration parameters on one page.

Besides manipulating the generation of WikiNames, LdapContrib does take care to keep a mapping stable between cache refreshing. When a new cache.db is build up, it takes the old one under consideration and keeps the already existing login => WikiName mapping.

Use the rerfesh=force parameter to really compute new WikiNames from scratch, that is without keeping the mapping stable.

Given these two features already existing in the Foswiki version of LdapContrib, I don't think that the described feature to have YACP (yet another config parameter) is required.

Please try to make use these existing features first to achieve the desired behavior. Let me know if that worked out on Foswiki. I could however imagine that RewriteWikiNames isn't there in the old TWiki version of LdapContrib unless somebody ported it over recently.

-- MichaelDaum - 28 Jun 2013

---

Hi Michael, and thanks for your input!

RewriteWikiNames has been ported indeed, so this is a potential solution. Our problem with using that is the fact that the TWikiUsers topic today defines about 12000 mappings between TWikiNames and logins, which would be a bit drastic to put in LocalSite.cfg for my taste. We could of course put everything in RewriteWikiNames, run the cache (either mode forced or when the cache is empty), and then take the rules out of the file again.

- - -

Regardless of that choice, the WikiName belonging to the users not longer present in the LDAP server has to be cached somehow, both because the issue JulianLevens is talking about (seeing JulianLevens and not jlevens), and that these WikiNames still may have access rights delegated to them, which may not be acceptable since future users can capture one of these, getting rights he shouldnt have. These are users that does not have a valid DN, and potentially no email address stored (depending on what the login is).

In our case we have stored them like WikiName <-> USERDOESNOTEXIST. What I am thinking of doing is inserting records like these before refreshUsersCache is run, whenever the cache is empty or mode is set to forced:

Once: $data->{"DN2U::placeholderdn"} = "placeholderuser"; $data->{"U2DN::placeholderuser"} = "placeholderdn";

And many times (to make sure these WikiNames are not taken when cacheUserFromEntry is run on the ldap entries) $data->{"U2W::placeholderuser"} = $wikiName; $data->{"W2U::$wikiName"} = "placeholderuser";

("placeholderuser" would ofcourse be replace by the actual username if we had this)

Any input? It could be that this use case is too narrow for general use and would have no interest outside our wiki..

---

-- TerjeNessAndersen - 28 Jun 2013

Actually, strike what I said about RewriteWikiNames. Because:

Lets say we have these 5 login <> WikiName relationsships stored in our Wiki installation:

• jsmith => JamesSmith
• john => JohnJohnson
• rwill => RobWilliams
• mbrown => MichaelBrown
• williamj => WilliamJones

All of these users have selected a WikiName manually during registration so far.

This is their real name stored in LDAP, which will be used for the generation of the WikiNames

• James Smith
• John Davis Johnson
• Robert Garcia Williams
• Michael Rodriguez Brown
• William Wilson Jones

Now, if we want to use RewriteWikiNames, we would have to first run LdapContrib on all logins (with the RewriteWikiNames setting empty), in order to find out what WikiNames LdapContrib generates for these users, namely:

• JamesSmith
• JohnDavisJohnson
• RobertGarciaWilliams
• MichaelRodriguezBrown
• WilliamWilsonJones

Then we would know what to put in RewriteWikiNames:

$Foswiki::cfg{Ldap}{RewriteWikiNames} = {
    'JohnDavisJohnson'        => 'JohnJohnson',
    'RobertGarciaWilliams'    => 'RobWilliams',
    'MichaelRodriguezBrown'   => 'MichaelBrown',
    'WilliamWilsonJones'      => 'WilliamJones',
};

For me, this seems like a worse way of doing things, wouldn't you agree? Or am I missing something?

-- TerjeNessAndersen - 28 Jun 2013

Is the login name still the same?

-- MichaelDaum - 28 Jun 2013

In our instance, the login, aka $Foswiki::cfg{Ldap}{LoginAttribute} stays the same yes. If a physical user returns from SSO login with a different login name, he will be given (or asked for, like today) a new WikiName, as the system regards this as a new user.

People have been able to set their own WikiName in registration to whatever until now, and this relationship will need to be preserved in the LdapContrib cache.

-- TerjeNessAndersen - 28 Jun 2013

I understand better now. The problem is that you need better support when migrating from TopicUserMapping to LdapContrib. You need support to keep the old mappings while extending it from there on using the mechanisms provided by LdapContrib.

In case you had {AllowLoginName} you don't need another topic to preserve the mapping. It is already there in Main.WikiUsers..

If you didn't have {AllowLoginName} in place, then you are in more problems as there's no easy way to find out which old WikiNames map to new login+wikinames.

-- MichaelDaum - 01 Jul 2013

Yes, sorry for explaining myself badly!

• We have {AllowLoginName}, and it's set to true.
• We are also currently using TopicUserMapping, with the Main.WikiUsers topic, where mappings between login and WikiName are stored. It was this topic I was referring to when i said "A topic containing existing WikiName <-> login mappings can be set, if wanted.".

But in order to preserve the mappings set in this topic (Main.WikiUsers), some logic has to be added right? If this relationship is set in Main.WikiUsers for jdoe:

   * JohnCoolDoe - jdoe

The day comes to install LdapContrib, {UserMappingManager} is set to LdapUserMapping instead of TopicUserMapping. Now, when the LdapContrib cache is first built, if {Ldap}{WikiNameAttributes} = 'displayName' and the displayName (in LDAP) is "John Doe" for login "jdoe", then the cache entry should be:

 U2W::jdoe
 JohnDoe

And the next time "jdoe" logs in, he has the WikiName JohnDoe right? (And not JohnCoolDoe).

So as you say it, I'm trying to find a way to preserve the mappings stored by TopicUserMapping when switching to LdapUserMapping. As I see it, there is no such support today, and I can imagine that others making this switch will be facing the same problem also. Or what do you think?

-- TerjeNessAndersen - 01 Jul 2013

Most organizations go with an ldap setting right from the start. The current {RewriteWikiNames} is n't really addressing your needs. It is more covering name changes as they usually happen.

So from where you are, we'd trigger a new behavior using:

$Foswiki::cfg{Ldap}{PreserveTopicUserMapping} = 1;

which would take the WikiName for a login as specified in the Main.WikiUsers topic.

That's a change to Foswiki::Contrib::LdapContriub::refreshUsersCache. In case {PreserveTopicUserMapping} is enabled, it should call cacheUserFromEntry() with an explicit WikiName.

  my $topicWikiName;
  $topicWikiName = <get it from a topic based user mapping hash> if $Foswiki::cfg{PreserveTopicUserMapping};

  $args{callback} = sub {
    my ($ldap, $entry) = @_;
-    $this->cacheUserFromEntry($entry, $data, \%wikiNames, \%loginNames) && $nrRecords++;
+    $this->cacheUserFromEntry($entry, $data, \%wikiNames, \%loginNames, $topicWikiName) && $nrRecords++;
  };

-- MichaelDaum - 01 Jul 2013

Seems good, I have one issue though:

Prior to implementing LdapContrib (when using TopicUserMapping), there might well be users on the LDAP server that doesn't have a WikiName yet.

If cacheUserFromEntry is first run where the WikiName is undefined (login is not defined in $topicWikiName), then a WikiName is calculated and cached in the temporary cache.

Now, if cacheUserFromEntry is run a second time, now with WikiName defined (login is defined in $topicWikiName), and this WikiName conflicts with the first run, then this WikiName is going to be renamed as I read the code, not following the rules of the Main.WikiUsers topic.

I have currently solved this by not caching users "as you go", when performing the LDAP search. Instead, I split the users into two arrays, 1) the users returned from LDAP which has an entry in Main.WikiUsers. 2) the users returned from LDAP which does not have an entry in Main.WikiUsers.

Then the users which has an entry in Main.WikiUsers is processed first, and then the users that don't, ensuring that all the WikiUsers in the first list wins in a clash.

What do you think about this approach?

Also, I hope that this feature could be of use to the LdapContrib plugin. My motivation for discussing this here is not mainly to get help on how to implement this in my own environment, but to discuss a solution that eventually could be made public for the use of others.

-- TerjeNessAndersen - 02 Jul 2013

Your suggestion makes totally sense giving the topic users a higher precedence in case there is a name clash.

Whatever you find out that fixes the issue you have and that you solved in your environment, it would be great to see your work in improving things being contributed back as your problem seems common for other people migrating. People not contributing their local changes upstream will have a hard time when upstream moves on without their own changes on board.

So I'd like to encourage you to provide a patch to be merged upstream.

Thanks in advance.

-- MichaelDaum - 02 Jul 2013

Allright, will do!

The only problem now is how to cache login <-> WikiName mappings belonging to logins deleted on the LDAP server. These logins does not get a "hit" when queuing the LDAP Server, so they will have to be processed separately. Since these WikiNames could have rights assigned to them, it's important to prevent them from being snatched by active logins/users.

This is the current plan on how to solve this:

• add the subfunction addIgnoreWikiName that puts the WikiName in comma-separated list in $data->{IGNWNAMES}
  • after splitting the valid logins into two arrays as mentioned in my last post, a nice side-effect of this is that we now know which logins that didn't get put in these two arrays (the deleted users).
  • to not have to take the Main.WikiUsers topic into consideration every time the cache is refreshed (only when the cache is empty or mode is refreshed), $data->{IGNWNAMES} would have to be copied over to %tempData, somehow like this:

   my %tempData;
   my $tempCache = tie %tempData, 'DB_File', $tempCacheFile, O_CREAT | O_RDWR, 0664, $DB_HASH
      or die "Cannot open file $tempCacheFile: $!";
+  $tempData{IGNWNAMES} = $this->{data}->{IGNWNAMES} if $this->{data}->{IGNWNAMES} && $refreshMode < 2;

• In cacheUserFromEntry, the following logic is added:

-  # 5. check if this dn maps to a wikiName already in use by another dn before
-  my $prevDN = $this->getDnOfWikiName($wikiName) || '';
+  # 5. check if this dn maps to a wikiName already in use by another dn before, or if the wikiname should be ignored.
+  my $prevDN = $this->getDnOfWikiName($wikiName) || $this->getIgnoredWikiName($wikiName) || '';

I have not tested this approach yet, but any feedback is of course welcome

-- TerjeNessAndersen - 02 Jul 2013

That's a different problem. Try to fix both in separate patches.

-- MichaelDaum - 02 Jul 2013

Any progress on this one? Patches welcome.

-- MichaelDaum - 22 May 2014

I've implemented this feature now from scratch as it is almost too much of a low hanging fruit:

• Parameter {UserMappingTopic}: if non-empty will read a pre-defined user mapping, e.g. from WikiUsers
• in cacheUserFromEntry it looks up this mapping if required and takes values from there instead of using the {WikiNameAttributes}
• then things proceed as usual.

Terje, I've looked at your modifications done on the TWiki project. Man, your version grew another 1300 lines of code. Even the rest is quite different by now. I can't possibly see a way how to merge both.

One important missing feature in your version is referrals and links in LDAP directories. Your code can't deal with it atm.

-- MichaelDaum - 23 May 2014

Hi Michael,

Yes, unfortunately the code grew a lot. It needs a proper refactoring. I would also like to go from BerkeleyDB to a relational database. I find it too easy to corrupt the database with BerkeleyDB, since there are so many different keys which might rely on one another.

At CERN, we're in the process of upgrading to TWiki 6 now. After that, we might check out Foswiki, and maybe do some further enhancements to LdapContrib as well. Let's see.

-- TerjeNessAndersen - 29 Aug 2014

ALL of the user management should be backed up by a real DB. LdapContrib will just feed that DB with its IDs. That is: the move to a real DB should not be an ldap-only thing.

-- MichaelDaum - 29 Aug 2014

Commenting is disabled while not logged in