Item13076: Bug: LdapContrib fails to create cache.db on FreeBSD

pencil
Priority: Urgent
Current State: Closed
Released In: n/a
Target Release: n/a
Applies To: Extension
Component: LdapContrib
Branches: master
Reported By: ChristophLarsen
Waiting For:
Last Change By: ChristophLarsen
Dear All, My setup: FreeBSD jail with Foswiki, running via FastCGI and Nginx All well, till I installed the LDAPContrib plugin, as LDAP authentication is urgently required. Entered the correct LDAP config; tools/ldpatest works fine, too. Upon changing {UserMappingManager} to Foswiki::Users::LdapUserMapping OR {PasswordManager} to Foswiki::Users::LdapPasswordUser, the system throws: Error tieing cache file path: Inappropriate file type or format After some disucssion on IRC, the core of the problem seems to be that LDAPContrib creates working/working_areas/LdapContrib/cache.db and cache.db.lock with the right permissions, but cache.db is persistently zero-length. This makes LDAPContrib complain. No solution in sight, db48 versus db5 - no difference. This bug renders Foswiki useless, as LDAP for our OpenLDAP-bases CAS is a preconditon for deployment. Any thoughts, ideas? Thanks a lot, indeed!

-- ChristophLarsen - 05 Nov 2014

LdapContrib is used in many Foswikis day to day. Never seen this error. We tried to get more info on IRC but none of it looked suspicious.

-- MichaelDaum - 05 Nov 2014

Thanks, Michael, appreciated. But the issue has to get solved, or Foswiki is not it... which woud be a serious pity. The core question remains: Why is cache.db created with zero length? How is it created? How is the tie performed? I have tried different versions of DB_File, DB_File_Lock and BerkeleyDB with zero joy. Alternatively, can it be a FreeBSD jail-specific issue? I have BerkeleyDB running in other jails, no problems there, nor with LDAP client. A big thank you! Chris

-- ChristophLarsen - 05 Nov 2014

One thing I noted - thinking further along the line why LDAPContrib does create cache.db, but only with zero length - is that LDAPContrib's way of creating cache.db does not seem to respect the {RCS}{dirPermission} and {RCS}{filePermission} settings in SiteLocal.cfg. In contrast, cache.db.lock follows the designated permission rules. This is probably a longshot, but we should look into how cache.db is created. With the set-up and the configuration being well in order, this issue meets the definition requirements for a bug - possibly because I am not running Linux, but *BSD.

-- ChristophLarsen - 06 Nov 2014

{RCS}{*Permissions} don't apply here as these files are not part of the RCS store. Those permission settings are only used when creating webs and topics in .../data/... and .../pub/... , not temporary files located in ..../working/work_areas/LdapContrinb/...

Looking at the code, it actually does create an empty cache.db file by purpose when it does not exist yet. So could you please say at which line exactly the code fails. This is in Foswiki::Contrib::LdapContrib::tieCache(): which of the two tie operations fail for you?

Could very well be that this is a *BSD problem. What kind of filesystem does your working/work_areas/... directory live on?

-- MichaelDaum - 06 Nov 2014

Here is the full log output that I get after deleting all files in working/work_areas/LdapContrib/, and issuing ./view refreshldap=on as the user that normally runs Foswiki:
./view refeshldap=on Error tieing cache file /usr/local/fwiki/synalinq/working/work_areas/LdapContrib/cache.db: Inappropriate file type or format at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 792. at /usr/local/lib/perl5/5.16/CGI/Carp.pm line 379, line 751. CGI::Carp::realdie('Error tieing cache file /usr/local/fwiki/synalinq/working/wor...') called at /usr/local/lib/perl5/5.16/CGI/Carp.pm line 468 CGI::Carp::die('Error tieing cache file /usr/local/fwiki/synalinq/working/wor...') called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 792 Foswiki::Contrib::LdapContrib::tieCache('Foswiki::Contrib::LdapContrib=HASH(0x804cfd468)', 'read') called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 829 Foswiki::Contrib::LdapContrib::initCache('Foswiki::Contrib::LdapContrib=HASH(0x804cfd468)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 322 Foswiki::Contrib::LdapContrib::getLdapContrib('Foswiki=HASH(0x8033a07c8)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/LdapPasswdUser.pm line 60 Foswiki::Users::LdapPasswdUser::new('Foswiki::Users::LdapPasswdUser', 'Foswiki=HASH(0x8033a07c8)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/TopicUserMapping.pm line 66 Foswiki::Users::TopicUserMapping::new('Foswiki::Users::LdapUserMapping', 'Foswiki=HASH(0x8033a07c8)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/LdapUserMapping.pm line 53 Foswiki::Users::LdapUserMapping::new('Foswiki::Users::LdapUserMapping', 'Foswiki=HASH(0x8033a07c8)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Users.pm line 104 Foswiki::Users::new('Foswiki::Users', 'Foswiki=HASH(0x8033a07c8)') called at /usr/local/fwiki/synalinq/lib/Foswiki.pm line 1762 Foswiki::new('Foswiki', 'AdminUser', 'Foswiki::Request=HASH(0x803394f30)', 'HASH(0x8033923c0)') called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 306 Foswiki::UI::__ANON__() called at /usr/local/fwiki/synalinq/lib/CPAN/lib/Error.pm line 379 eval {...} called at /usr/local/fwiki/synalinq/lib/CPAN/lib/Error.pm line 371 Error::subs::try('CODE(0x80184d780)', 'HASH(0x8033a03a8)') called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 435 Foswiki::UI::_execute('Foswiki::Request=HASH(0x803394f30)', 'CODE(0x803394a98)', 'command_line', 1, 'view', 1) called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 274 Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x803394f30)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Engine/CLI.pm line 53 Foswiki::Engine::CLI::run('Foswiki::Engine::CLI=HASH(0x801af1540)') called Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.

Error tieing cache file path: Inappropriate file type or format
Does this give any further clue on what could be wrong? Thank you so much! Chris

-- ChristophLarsen - 06 Nov 2014

Searching the web for "Inappropriate file type or format" shows that you are not alone with this problem. As far as I was able to get the reports out there, there is a mismatch between DB_File and Berkeley DB installed in your FreeBSD jail.

Please check the version of DB_File and Berkeley DB installed on your system.

-- MichaelDaum - 06 Nov 2014

Dear Michael, Thanks for you reply - good to get some more ideas! p5-BerkeleyDB comes from ports, and is version 0.54_1 DB_file is actually part of the Foswiki 1.1.9 distribution, and is 1.826. (I think it is part of the basic perl 5.16.3_11 distribution that comes with FreeBSD.) I have just upgraded BD_File using cpanm to version 1.831. There is now later version of BerkelyDB around, as it seems. Change to the user that is supposed to run the Foswiki instance, and issuing ./view refreshldap=on continues, however, to throw the infamous error. Let me investigate, outside Foswiki, the mismatch you mentioned above. I will revert asap. Thanks a lot, indeed! Chris

-- ChristophLarsen - 06 Nov 2014

Forgot to mention - the whole Foswiki instance is located on a local fs.

-- ChristophLarsen - 06 Nov 2014

There is a DB_file.pm in Foswiki distribution, but it's not the CPAN module. Foswiki ships with Foswiki::Cache::DB_File which in turn has a use DB_File. That one comes from the system. It's not included in our lib/CPAN/lib directory.

-- GeorgeClark - 06 Nov 2014

Update: My research on Perl + FreeBSD + BerkeleyDB + DB_File + "Error tieing cache file path: Inappropriate file type or format" hinted towards issues with NFS. I have, running Foswiki in one FreeBSD jail and my reverse proxy in another, deployed nullfs-mounting for Foswiki's /pub directory. Likewise, Foswiki's /data folder got nullfs-mounted via a big partition. I put everything into one partition, deleted the files in /working/work_areas/LdapContrib/, became the Foswiki user and ran ./view refreshldap=on. Alas, still no joy. Are there any FreeBSD users out there running Foswiki??? I am extremely reluctant to run the Foswiki version packaged with FreeBsd, as this is version 1.1.5, while all subsequent stable versions have been, AFAIK, bug and security fixes. I am running out of ideas. Any recommended versions for BerkeleyDB and DB_File? Thanks a lot! Chris

-- ChristophLarsen - 06 Nov 2014

@George, thank you, only saw your comment now. Yes, I can confirm: There is ./lib/Foswiki/Cache/DB_File.pm calling the system's DB_File. My system's DB_File is in /usr/local/lib/perl5/5.16/mach/DB_File.pm, version version 1.818, coning from ports. Anything wrong here?

-- ChristophLarsen - 06 Nov 2014

And, yes, after having rules out any issues surrounding nullfs, I have looked into this one further: http://unix.derkeiler.com/Mailing-Lists/FreeBSD/questions/2006-05/msg00325.html Essentially, I would have to know, which version of BerkeleyDB LDAPContrib has been built with originally, and from there, we can look into the best version of BerkeleyDB and DB_File. Strangely enough, Perl itself does not seems to be able to get this right (?). To start with, are you happy with perl 5.16?

-- ChristophLarsen - 06 Nov 2014

Best would be too cook up a simple test.pl that imitates the code in Foswiki::Contrib::LdapContrib::tieCache(). Then run it once or twice to see whether your BerkeleyDB/DB_File combo on FreeBSD runs fine. I've seen similar example code on the net doing exactly that to repro the error.

If you are using packages of your distro only with nothing else that could be in the way and this test.pl still throws the same error, well then your installation is busted for whatever reason. There are a lot of other packages that should fail to run properly on your setup with a similar error as DB_File is quite common for little things to store persistently (spamassassin for one I guess).

-- MichaelDaum - 06 Nov 2014

Spamassassin works like a charm, Michael :-(. In fact, the system is a pretty seasoned, highly established evironment.

-- ChristophLarsen - 07 Nov 2014

Mind you, I always install from ports, i.e. compile everything from source - for added stability (and speed). I would have been alarmed during the compilation process, is sth had gone wrong. Which Perl version do you usually work with?

-- ChristophLarsen - 07 Nov 2014

I am using plenv compiling a perl + modules for local Foswiki use. Latest stable is 5.20.1. I had no such DB_File problems since forever. LdapContrib started 2006, so it has seen quite some different perl versions along the way.

Here's a test_db_file.pl. Does it croak like LdapContrib?

-- MichaelDaum - 07 Nov 2014

Thank you so much, Michael. Let me test...

-- ChristophLarsen - 07 Nov 2014

Your test_db.pl works as smoothly as silk. Creates a nice-looking /tmp/cache.db plus lock file. Not even the hint of a croaking sound...

-- ChristophLarsen - 07 Nov 2014

For the fun of it (as I feel we are running out of options), I upgraded perl from 5.16 to 5.20. Does not make any difference, unfortunately.

-- ChristophLarsen - 07 Nov 2014

Try a fileName so that the test scripts writes to .../working/work_areas/LdapContrib/some.db and execute the script from within the Foswiki jail.

-- MichaelDaum - 07 Nov 2014

Thanks again, Michael.

Here is what I did:

(1) I changed filename inside test_db_file.pl to: my $fileName = "/usr/local/fwiki/synalinq/working/work_areas/LdapContrib/some.db";

(2) I copied test_db_file.pl into ./bin/, adjusted permissions to 700, for [instance_user]:www.

(3) I then issue as root: su - [instance_user]

(4) Changed to ./bin, and issued as [instance_user]: ./test_db_file.pl

RESULT: The output was satisfactory: creating /usr/local/fwiki/synalinq/working/work_areas/LdapContrib/some.db re-opening /usr/local/fwiki/synalinq/working/work_areas/LdapContrib/some.db

I know yo are going to hate me for that :-/. Still no useful diagnostic output.

-- ChristophLarsen - 07 Nov 2014

So, in summary, the diagnostics so far have not provided any clue as to why we have this issue.

-- ChristophLarsen - 11 Nov 2014

On more output from the following crontab entry after an experimental upgrade to perl 5.20:
cd /usr/local/fwiki/synalinq/bin && ./view refreshldap=on Main/WebHome >/dev/null
CGI::Carp will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at ./view, line 9. [Wed Nov 12 00:05:00 2014] CGI.pm: CGI will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at /usr/local/fwiki/synalinq/lib/Foswiki.pm, line 49. [Wed Nov 12 00:05:00 2014] Util.pm: CGI::Util will be removed from the Perl core distribution in the next major release. Please install it from CPAN. It is being used at /usr/local/lib/perl5/5.20/CGI.pm, line 29. [Wed Nov 12 00:05:00 2014] view: Useless use of greediness modifier '?' in regex; marked by <-- HERE in m/^([0-9]+)\.([0-9]+)\s+date\s+(\d\d(\d\d)?(\.\d\d){5}? <-- HERE );$/ at /usr/local/fwiki/synalinq/lib/Foswiki/Store/VC/RcsLiteHandler.pm line 304. Error tieing cache file /usr/local/fwiki/synalinq/working/work_areas/LdapContrib/cache.db: Inappropriate file type or format at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 792. at /usr/local/lib/perl5/5.20/CGI/Carp.pm line 379, line 751. CGI::Carp::realdie("Error tieing cache file /usr/local/fwiki/synalinq/working/wor"...) called at /usr/local/lib/perl5/5.20/CGI/Carp.pm line 468 CGI::Carp::die("Error tieing cache file /usr/local/fwiki/synalinq/working/wor"...) called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 792 Foswiki::Contrib::LdapContrib::tieCache(Foswiki::Contrib::LdapContrib=HASH(0x803e88ab0), "read") called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 829 Foswiki::Contrib::LdapContrib::initCache(Foswiki::Contrib::LdapContrib=HASH(0x803e88ab0)) called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 322 Foswiki::Contrib::LdapContrib::getLdapContrib(Foswiki=HASH(0x803acfe28)) called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/LdapPasswdUser.pm line 60 Foswiki::Users::LdapPasswdUser::new("Foswiki::Users::LdapPasswdUser", Foswiki=HASH(0x803acfe28)) called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/TopicUserMapping.pm line 66 Foswiki::Users::TopicUserMapping::new("Foswiki::Users::LdapUserMapping", Foswiki=HASH(0x803acfe28)) called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/LdapUserMapping.pm line 53 Foswiki::Users::LdapUserMapping::new("Foswiki::Users::LdapUserMapping", Foswiki=HASH(0x803acfe28)) called at /usr/local/fwiki/synalinq/lib/Foswiki/Users.pm line 104 Foswiki::Users::new("Foswiki::Users", Foswiki=HASH(0x803acfe28)) called at /usr/local/fwiki/synalinq/lib/Foswiki.pm line 1762 Foswiki::new("Foswiki", "AdminUser", Foswiki::Request=HASH(0x803ac59d8), HASH(0x803aba078)) called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 306 Foswiki::UI::__ANON__() called at /usr/local/fwiki/synalinq/lib/CPAN/lib/Error.pm line 379 eval {...} called at /usr/local/fwiki/synalinq/lib/CPAN/lib/Error.pm line 371 Error::subs::try(CODE(0x80184e870), HASH(0x803acfa08)) called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 435 Foswiki::UI::_execute(Foswiki::Request=HASH(0x803ac59d8), CODE(0x803aaba80), "view", 1, "command_line", 1) called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 274 Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x803ac59d8)) called at /usr/local/fwiki/synalinq/lib/Foswiki/Engine/CLI.pm line 53 Foswiki::Engine::CLI::run(Foswiki::Engine::CLI=HASH(0x80226d930)) called
Is there any chance to solve this conundrum? I do not want to waste anybody's time, but the issue raised by me sonds like a pretty obvious bug, and is a complete showstopper. If there is no further input, Foswiki will be disqualified from our pilot serving a national, possibly even regional base. Thanks a lot, indeed, for your help. Chris

-- ChristophLarsen - 12 Nov 2014

Sorry Chris, but there is not much we can help you with here in the scope of community support. Note that there always is the option to hire one of the listed WikiConsultants. This would allow you to have somebody else look at it hands on.

Btw, do not use RcsLite: it is busted.

-- MichaelDaum - 12 Nov 2014

Dear Foswiki Community, Allow me to express my concern with the above reply by Michael. We have, as you can see from the above, encountered an issue that very much looks like a bug, though I certainly have not stopped searching for any possible causes in my own yard. It may be a FreeBSD or *BSD thing, but I blindly assume that the Foswiki WANTS to see Foswiki deployed on *BSD, too. (The FreeBSD Foswiki port is, BTW, old.) I have struggled, and succeeded to make Foswiki fly with Nginx, squeezed out every bit of juice for speed, and have announced that I am very happy ot share my configuration and setup. HOWEVER, to refer a bug to paid services is, to put it mildly, just a BAD MOVE, and, I am sure you will agree, completely in contradiction to the spirit of **FOS**wiki. After all, I am not asking for a new feature, but for a very, very, very common deployment option, a.k.a. OpenLDAP, that should just work. Admittedly, the route Foswiki takes with OpenLDAP looks complex - I have been using OpenLDAP as a central authentication service for all my applications, and never before did I have to leverage BerkelyDB, etc. So, undue complexity MAY be the cause, yet I am not a Perl expert to comment with authority. However I do state with authority that you just cannot refer bugs to paid services, sorry. This is against the spirit of your very own split from twiki, and your commitment to FLOSS. I am actually glad that it came to this before I have been able to load any data, and therefore have not wasted even more time. Extremely unhappy.

-- ChristophLarsen - 13 Nov 2014

Sorry to read this Chris, really.

The Foswiki community has already given you a lot of support on your issues, here, on irc, via email etc. We'd be glad to see your problems being addressed and make you YAGFU (Yet Another Glad Foswiki User).

However, you seem to be using a "somehow strange setting" in a way that is beyond the scope of exchanging information via irc, pastebin, foswiki.org, email and whatever channel we already used extensively with you.

I can't say more than "strange" because - as far as I was able to get it - your platform is not strange per se, nor your desire to integrate it into LDAP. Foswiki+LDAP is very common way the system is being used for ages in corporate environments successfully.

So there must be something different on your platform that you experience these problems. We have not been able to figure out what it is. Sorry for that!

Pointing you to hire professional help seems to be an optional sensible next step for you as it clearly needs somebody to look at it hands on. Please bear in mind that open source support is by nature limited as we can only give you written advice over the net without actually looking at your server ourselves. So there are innate limits to what we can do here. Beyond that you need somebody willing to help you out more in depth and potentially signing an NDA with your organization ... something only a professional consultancy is able to do.

Offering for-money services to fix bugs in open source products isn't uncommon at all and by no means against the FLOSS idea. The contrary is the case: "Fix & Enhance" services are what makes the ecosystem running.

See for instance Fix My Qt Bug! offered by KDAB recently or Qt.io's services.

Long story short, you have three options:

  1. hire somebody to fix the bug for you; think of it as a donation to those people making a living from Foswiki
  2. fix the bug yourself; FLOSS lives from volunteers like you to improve software. Patches welcome!!!
  3. neither of above; the bug will remain unfixed for you

Your choice.

-- MichaelDaum - 13 Nov 2014

Michael, I don agree. Foswiki is capable, but severely limited by its anachronistic insistence on Apache, CGI and the lot. This has created some spahetti-code that makes is quite evidently difficult to adapt to more up-to-date web server frameworks. You mentioned, for instance, RcsLite being busted. However, with FastCGI, you have no alternative. So, we have one operational regression after the other, all due to the above. I am sure that the issue above is related to the same, if you like, strategic, shortcomings. The documentation reflects te same, and I was about to add substantiallly to the same. However, with this attitude, I serious concerns regarding the flexibility and architectural openess of Foswiki, and have herewith stopped the feasibility testing. Thank you for speeding up the process, albeit with a negative outcome.

-- ChristophLarsen - 13 Nov 2014

Christoph, I have to jump in here with a few observations.

  1. We do NOT have anachronistic insistence on Apache, CGI and the lot. Foswiki works just fine with FastCGI or Mod_perl, and it works and is tested under Nginx, and Lighttpd as well as apache. We are still looking for a volunteer to support IIS.
  2. As far as CGI goes, Foswiki roots go back to 1998. So it's an evolution. From http://openhub.net, Foswiki "... has had 25,076 commits made by 76 contributors representing 4,292,065 lines of code." and "... took an estimated 1,265 years of effort (COCOMO model) starting with its first commit in October, 2008 ending with its most recent commit 1 day ago" (Though a lot of that is jquery) We don't insist on CGI, but there is a bit of a codebase that needs volunteers to help modernize.
  3. We also don't insist on Linux. We test on Windows and various flavors of linux, though mainly with apache. But Foswiki.org runs on FreeBSD, so that's covered as well. However our public site on freebsd doesn't use LDAP, so we just don't have that particular combination available.
  4. RcsLite being busted is probably a bit of an overstatement. In Foswiki 1.1.9, we invoke RCS a bit too often, and RcsLite is slow, but mainly with large binary files. (It expands history in memory). It's also due to some overzealous validation of the revision history, as well as some bugs. It's improved in the upcoming 1.2.0 There is also a hotfix for 1.1.9 that can improve RCS performance a bit. So I'd say better advice is to use it with caution. Foswiki.org runs with FastCGI and RcsLite, so it's certainly not that broken.
  5. Also in upcoming 1.2, we've made a PlainFile store the default store for new sites. No RCS involved. So these things are being addressed, just not as fast as we would like.
  6. For FastCGI, RcsWrap is also usable. You have the overhead of the fork, but it is much less severe than on mod_perl where the fork applies to the entire apache install.
  7. Also in the upcoming 1.2, volunteers have completely rewritten configure. It no longer requires CGI either. And we are testing install of 1.2 on Nginx / fcgid "out of the box" to ensure a better installation experience. With 1.2, the experience we are testing towards is:
    • Unzip foswiki
    • Configure webserver to point to correct file system locations
    • Visit your default View URL, short, long, however you want it.
    • Foswiki bootstrap code detects critical settings, "just works" and provides you with a link to configure to check & save the discovered configuration.

Bottom line though is unfortunately, you encountered a bug. It appears to be rather obscure, in that there have been multiple unsuccessful attempts to diagnose it. It's going to need some in-depth debugging, possibly with embedded print statements, or even running foswiki under a debugger. We just don't appear to have the right combination of servers and configuration to recreate your issue. Without a recreation it's pretty difficult to fix it.

We are all volunteers, with very limited budgets on what we have for foswiki test infrastructure. There is no corporation behind Foswiki, beyond our small Foswiki Association. You appear to be very knowledgeable, and Foswiki succeeds because of knowledgeable volunteers. So best advice I can offer is to suggest that you do some more debugging, and contribute a fix for the issue. If you can help narrow it down, I'm sure volunteers will be willing to help fix it.

But if your expectation is that we go out and buy hardware, build test environments to match your configuration, or send someone off to your site to help you debug, or shift our volunteer working hours to be online during your workday and interact with IRC, that's just not reasonable.

I'm sorry you feel that you need to find some other solution. Foswiki is really and excellent, mature platform with a long track record of good compatibility of new releases. But all software has bugs, and with open source, it's the users & volunteers who help fix the issues. If you can find something better, we wish you luck, and if you want to help us, we welcome your participation.

-- GeorgeClark - 13 Nov 2014

Dear George, Thank you for your long reply. As a matter of fact, I agree with everything your say, only got my fur bristled when I got the former response of "go out and pay to get that bug fixed". If you are in a position where you have to select suitable software for a huge deployment, where, over time, lots of people wil lget locked in by their sheer effort, you understand that the abve statement raises all avaiable red flags. However, your approach of a collaborative effort, and response is exactly what drives good-quality FLOSS. Testers (even unvoluntary ones like me, who encounter a pesky bug) are essential and, yes, testing is work in itself. I am encouraged to see you defusing the "RcsLite is busted' statement, as this would have really been a bit of a FastCGI killer. (I do have to look into RcsWrap, admittedly.) Of course, I would never even think of demanding that you buy hardware and provide bespoke service - but we should try to get thngs fixed, as this drives Foswiki forward pro bono publico. I am more than happy to provide you with that kind of testbed - my FreeBSD jail is just built for that purpose. So, with this fortunate turn of events, I am happy to continue playing guniea pig. Maybe you can enlkighten me a bit as to what debug knobs to twist and turn, as I am, alas, not a Perl expert. Again, thank you all for this constructive discussion. Chris

-- ChristophLarsen - 14 Nov 2014

I have just finished a complete re-install of the FreeBSD jail running Foswiki - and the error could be reproduced, as soon as I set {UserMappingManager} to Foswiki::Users::LdapUserMapping, and {PasswordManager} to Foswiki::Users::LdapPasswdUser.

-- ChristophLarsen - 14 Nov 2014

The installation procedure has been fully documented. If you wish to have a look (and also benefit from a reasonably detailed documentation of Foswiki with FastCGI, Ngnix, etc.), let me know where we can share it.

-- ChristophLarsen - 14 Nov 2014

For your docs, you could create a topic on the Support web (I suggest FoswikiOnFreeBSDandNginx , and add a link to InstallingOnSpecificPlatforms

As far as debugging, I don't have access to any LDAP infrastructure to explore with. For some general suggestions:
  • If you have not done so, create a bin/LocalLib.cfg. (Copy example from bin/LocalLib.cfg.txt. Only change needed is to remove the comment (#) from the line: #$ENV{FOSWIKI_ASSERTS} = 1; That enables additional debug checks in Foswiki code and can sometimes reveal things. It also causes stack traces to go to the browser instead of the logs.
  • Try running the failing command from the shell. cd to your bin directory, and run ./view -topic SomeTopic ... Running outside of the browser / CGI environment can reveal things sometimes.
  • From there, I'd be probably adding print statements to code, both Foswiki and perl modules, trying to get a better picture of what is failing. Or use the perl debugger to try to find and step into the failure to better understand it.

There are a number of admins using with LDAP, who come and go on IRC. It's a bit of the luck of the draw to catch them, so working from home at odd hours helps to find support. As we said your issue appears to be some combination of software that we just haven't encountered.

-- GeorgeClark - 14 Nov 2014

Could you please try this patch, please:

--- data/foswiki-extensions/lib/Foswiki/Contrib/LdapContrib.pm  (revision 1090)
+++ data/foswiki-extensions/lib/Foswiki/Contrib/LdapContrib.pm  (working copy)
@@ -781,7 +781,7 @@ sub tieCache {
    unless (-e $this->{cacheFile}) {
      $locking->{'mode'} = 'write';
      my %db_hash;
-    my $x = tie(%db_hash, 'DB_File::Lock', $this->{cacheFile}, O_CREAT, 0664, $DB_HASH, $locking);
+    my $x = tie(%db_hash, 'DB_File::Lock', $this->{cacheFile}, O_CREAT|O_RDWR, 0664, $DB_HASH, $locking);
      undef($x);
      untie(%db_hash);
      $locking->{'mode'} = $mode;

I just received it from Anton Yuzhaninov. Not tested it myself.

-- MichaelDaum - 14 Nov 2014

Dear Michael, Unfortunately, Anton's patch does not seem to do anything. The error output remains the same. Guess this would have been way too easy... frown, sad smile

-- ChristophLarsen - 17 Nov 2014

Dear George, Sorry my delay in responding - I have been travelling. I activated $ENV{FOSWIKI_ASSERTS} = 1 in ./bin/LocalLib.cfg, and got this after issuing ./bin/view ldapreset=on as the user that is supposed to run Foswiki:
./view ldapreset=on Error tieing cache file /usr/local/fwiki/synalinq/working/work_areas/LdapContrib/cache.db: Inappropriate file type or format at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 792. at /usr/local/lib/perl5/5.16/CGI/Carp.pm line 379, line 751. CGI::Carp::realdie('Error tieing cache file /usr/local/fwiki/synalinq/working/wor...') called at /usr/local/lib/perl5/5.16/CGI/Carp.pm line 468 CGI::Carp::die('Error tieing cache file /usr/local/fwiki/synalinq/working/wor...') called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 792 Foswiki::Contrib::LdapContrib::tieCache('Foswiki::Contrib::LdapContrib=HASH(0x804ca9678)', 'read') called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 829 Foswiki::Contrib::LdapContrib::initCache('Foswiki::Contrib::LdapContrib=HASH(0x804ca9678)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Contrib/LdapContrib.pm line 322 Foswiki::Contrib::LdapContrib::getLdapContrib('Foswiki=HASH(0x803755210)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/LdapPasswdUser.pm line 60 Foswiki::Users::LdapPasswdUser::new('Foswiki::Users::LdapPasswdUser', 'Foswiki=HASH(0x803755210)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/TopicUserMapping.pm line 66 Foswiki::Users::TopicUserMapping::new('Foswiki::Users::LdapUserMapping', 'Foswiki=HASH(0x803755210)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Users/LdapUserMapping.pm line 53 Foswiki::Users::LdapUserMapping::new('Foswiki::Users::LdapUserMapping', 'Foswiki=HASH(0x803755210)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Users.pm line 104 Foswiki::Users::new('Foswiki::Users', 'Foswiki=HASH(0x803755210)') called at /usr/local/fwiki/synalinq/lib/Foswiki.pm line 1762 Foswiki::new('Foswiki', 'AdminUser', 'Foswiki::Request=HASH(0x80374d978)', 'HASH(0x803747de0)') called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 306 Foswiki::UI::__ANON__() called at /usr/local/fwiki/synalinq/lib/CPAN/lib/Error.pm line 379 eval {...} called at /usr/local/fwiki/synalinq/lib/CPAN/lib/Error.pm line 371 Error::subs::try('CODE(0x80184d780)', 'HASH(0x803754dc8)') called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 435 Foswiki::UI::_execute('Foswiki::Request=HASH(0x80374d978)', 'CODE(0x80374d4e0)', 'command_line', 1, 'view', 1) called at /usr/local/fwiki/synalinq/lib/Foswiki/UI.pm line 274 Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x80374d978)') called at /usr/local/fwiki/synalinq/lib/Foswiki/Engine/CLI.pm line 53 Foswiki::Engine::CLI::run('Foswiki::Engine::CLI=HASH(0x802187c00)') called
In short: No change, really... Chris

-- ChristophLarsen - 17 Nov 2014

Chris, did you stop the webserver and fcgi backends, delete any zero-sized cache.db file from previous tries, patched LdapContrib.pm and then restarted everything?

Note that it is ./view refreshldap=on and not ldapreset.

-- MichaelDaum - 17 Nov 2014

Apologies - long international air travel does not exactly sharpen the mind. After deleting the stale cache.db and cache.db.lock file it does indeed not throw our old mysterious error any more. And of course, it is ./view ldapreset - sorry for that one. However, the previous error is re-reproducible again, as soon as I undo Anton's patch. So, we are getting somewhere. I will test more, and let you know! A huge thank you, Michael, George and - unbeknown to me - Anton! Chris

-- ChristophLarsen - 17 Nov 2014

I am glad to announce that Anton's patch did the trick. However, after having set up all the required (rather minimal) LDAP settings (which I have used with many other applications), nothing happens. The OpenLDAP log does not show any entries, as if the OprnLDAP server got never accessed. The firewall settings are of course alright, and OpenLDAP is happily accessed by other applications. I have deleted the cache.db and its lock, plus refreshed LDAP Foswiki settings beforehand, too. Any ideas? Thank you so much, Chris

-- ChristophLarsen - 21 Nov 2014

Add-on: I AM assuming that ContribLDAP supports user acconut generation on-the-fly, if there are valid OpenLDAP credentials. I certainly do not want to set up users twice - in OpenLDAP and in Foswiki. I am happy to allocate user roles from within Foswiki, as I use OpenLDAP for authenticaton, only.

-- ChristophLarsen - 21 Nov 2014

See Extensions.NewUserPlugin.

-- MichaelDaum - 22 Nov 2014

Thanks Michael, but it seems to me that simple, exteral LDAP authentication is made way to complicated in Foswiki. All I want is what other half-way decent wikis do out of the LDAP box: Add a few LDAP server and filter paramters, create the users in OpenLDAP, and have their uid, password and name created on-the-fyl within Foswiki, if they have valid OpenLDAP credentials. I fail to see the purpose ot the NewUserPlgun: I do not want to create user pages, and the documentation is patchy. I note that LdapNgPlugin is mentioned as obligtatory plugin, but I noted that all it does it to duplictate LdapContrib's fields on server details. Highly confusing. I also noted that LdapContrib does NOT retain the LFDAP server details I enter in bin/configure. The next day they are all reset to the default (useless) settings. In all honesty, and after having setup up a huge number of applications for external OpenLDAP authentication, I am allow to say that Fowiki's handling of the same is byzanthine. This is simply not user-friendly, obviously has lead to bugs (see above) and relys on overly complex dependencies, such as BerkeleyDB. The existing documentation, though it seems to be complete at first sight, does not cut it at all - it simply DOES NOT WORK! My server and system policy is just the opposite, read: Nginx instead of Apache. Qmail-LDAP instead of sendmail, etc. Great pity that. Again, my impression that foswiki is caught in some anachronistic spaghettis.

-- ChristophLarsen - 24 Nov 2014

  • LdapContrib: user mapping and LDAP infrasctructure
  • LdapNgPlugin: implement %LDAP macro to query LDAP in wiki apps, such as custom user profile templates
  • NewUserPlugin: create user profile pages on the fly when user log in via an external authentication scheme

I can't see any feature overlap.

The only problem I see that is worth this task item is fixing DB creation in FreeBSD as there seems to be an error in the semantics of perl's O_CREAT there. The requirement of an additional O_RDWR flag does not make much sense, but well. Using O_CREAT works fine on any other platform. Anyway, adding O_RDWR in addition to O_CREATE does not hurt, but helps FreeBSD.

-- MichaelDaum - 24 Nov 2014

I noted that you, Michael, closed this issue, even though there are clearly outstanding issuesw with FreeBSD. I take this as a clear example that you do not wish to pursue this issue further, and can ony conclude the FOSWIKI DOES NOT OFFER OEPNLDAP SUPPORT TO FREEBSD USERS. Thank you.

-- ChristophLarsen - 24 Nov 2014

Which other issues are there outstanding? I just made a new released based on Anton's patch. Please open another task item as soon as you come across another issue and clearly express your findings. Could you please lower the noise and concentrate on the real thing, will you?

-- MichaelDaum - 24 Nov 2014

Michael, As I have reported extensively above, the LdapContrib was first hampered by a bug that you encouraged my to seek commercial support for, and then for sheer non-functionality (as decscribed in detail above), upon which you decided to close the case. You will understand that I, for my part, can only conclude that Foswiki, OpenLDAP support and FreeBSD do not mix well, and that this kind of communication is nothing that could encouage the use of Foswiki with a large governmental or academic installtion in a resource-poor environment (that cannot afford a service contract). I thank you again for your help in getting this abundantly clear.

-- ChristophLarsen - 24 Nov 2014

ItemTemplate edit

Summary Bug: LdapContrib fails to create cache.db on FreeBSD
ReportedBy ChristophLarsen
Codebase 1.1.9
SVN Range
AppliesTo Extension
Component LdapContrib
Priority Urgent
CurrentState Closed
WaitingFor
Checkins LdapContrib:733fae798dd7
TargetRelease n/a
ReleasedIn n/a
CheckinsOnBranches master
trunkCheckins
masterCheckins LdapContrib:733fae798dd7
ItemBranchCheckins
Release01x01Checkins
I Attachment Action Size Date Who Comment
Install_Foswiki_—_synaLinQ.tar.bz2bz2 Install_Foswiki_—_synaLinQ.tar.bz2 manage 65 K 14 Nov 2014 - 10:58 ChristophLarsen Foswiki proper installation and configuration
Set_up_the_www_perl_Jail-_Your_Perl_Web_Application_Server_—.tar.bz2bz2 Set_up_the_www_perl_Jail-_Your_Perl_Web_Application_Server_—.tar.bz2 manage 63 K 14 Nov 2014 - 10:58 ChristophLarsen Setup documentation of FreeBSD Jail called www_perl, which houses Foswiki
test_db_file.plpl test_db_file.pl manage 756 bytes 07 Nov 2014 - 09:07 MichaelDaum  
Topic revision: r38 - 24 Nov 2014, ChristophLarsen
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy