Installing extensions breaks the configuration?

Hello,

I use ScientificLinux5. I wanted to install BlogAddOn plugin. Since I could not connect to foswiki.org via configure script, I decided to install it manually. I downloaded the archive, unpacked the files in the foswiki root directory and copied the files from _BlogAddOn/data directory to the web where I wanted to have it. It seemed to be ok, since configure now showed this extension in a separate line, however, there were new errors. One warning seemed to be not serious: data/_BlogAddOn had no WebPreferences.txt. However, the real problem was that comments.tmpl and CommentPlugin _installer could not be found in, respectively, ${FOSWIKI}/templates and ${FOSWIKI}/working/configure/pkgdata/, although both files were in those directories with the correct permissions and the ownership.

I could not correct the problem, thus I decided to reinstall foswiki, and copied the updated files from bin, lib, pub, and data. The system went back to normal with BlogAddOn present (not yet tested, if functional).

To pin down the problem, I downloaded CommentPlugin.tgz, unpacked the archive in the root catalog and ran `perl CommentPlugin _installer`. I ended up with the same problems as described earlier: files exist where they are sought, yet they are reported as "not found" by the configure script.

Could someone help in this matter?

Thanks,

Andrius

P.S. Later I tested BlogAddOn, and it seems to be working. So the question is, what became incorrect after the installation of the CommentPlugin, since I want to install more extensions.

---

The BlogAddOn has a broken Dependency file. It is using a version number (>=13313) for the CommentPlugin. That is actually from the TWiki svn repository numbering. The CommentPlugin for Foswiki 1.1 is at Rev: 8843 (2010-09-01).

You do not need to upgrade the CommentPlugin on 1.1. I'm not sure why the not-found files reported. I'm still trying to recreate that issue.

Opened Tasks.Item9804 about the bad dependency in BlogAddOn. I'll open a task against configure once I can recreate the issue.

If you have the exact text of the error messages about the missing comment templates, please post them here, along with information on exactly where they were reported. I've installed the BlogAddOn using your steps detailed above and did not encounter the exact issues you described.

Also when you downloaded CommentPlugin, what version did you get? I'm wondering if there was a window of time where you might have picked up the 1.0.x version accidentally.

-- GeorgeClark - 07 Oct 2010

A new version of the BlogAddOn has been uploaded to Foswiki.org. There are no operational changes. New version fixes:

• the bad dependency for CommentPlugin,
• adds a stub BlogAddOn.pm so configure can display the installed version.
• It also adds a WebPreferences topic to the _BlogAddOn web so that it will not generate errors in configure.
• And it moves the BlogAddOn settings into the Extensions tab.

Note that the 1.1 extension installer will not be able of installing this updated extension unless a WebPreferences.txt file is added to the data/_BlogAddOn directory. It can be copied from the data/_empty directory.

Tasks.Item9805 documents the issue with the extension installer failing to install files into an "incomplete" web.

---

1. I tried to recreate the problem once again. So I downloaded foswiki 1.1 and installed it into a separate directory, changed file ownership, created the updated config file for httpd, restarted the daemon, launched configure and saw everything mainly ok, once GeneralPathSettings were set. (The remaining error was e-mailing settings.) Then I ran `perl CommentPlugin _installer`, answered all as 'yes' (to reinstall CommentPlugin of version 8843 from 09-01). The installer reported no problems. I changed the ownership of the directories, restarted configure and saw the errors that I reported earlier:

Configuration states	`ls -la`
Error: /var/www/foswiki/Foswiki-1.1.0-20101007/templates/comments.tmpl cannot be found	-r--r--r-- 1 apache apache 160 Oct 4 19:25 /var/www/foswiki/Foswiki-1.1.0-20101007/templates/comments.tmpl
Error: /var/www/foswiki/Foswiki-1.1.0-20101007/working/configure/pkgdata/CommentPlugin_installer cannot be found	-rwxrwx--- 1 apache apache 4745 Oct 4 19:25 /var/www/foswiki/Foswiki-1.1.0-20101007/working/configure/pkgdata/CommentPlugin_installer

2. To test the updated BlogAddOn. I removed previous files (the fresh and free from user data installation), installed the foswiki, configured it. Unpacked the new version of the BlogAddOn (#9527), ran `perl BlogAddOn _installer`. The installer asked, if I wanted to reinstall the BlogAddOn (I guess it detected the files, present after untarring of the archive), I agreed, and ended up with BlogAddOn _installer not found in the 'working' directory. Yet BlogAddOn is listed in the `Extensions` lib, although it is not present in the list of 'enabled plugins'. The BlogAddOn skin is listed in the list of available skins in foswiki.

3. Since the suspicion fell on the running the perl script, I repeated the steps to start from scratch (I became good at this). Then I did exactly as the instructions say. I unpacked the BlogAddOn archive, changed the ownership of files and ended up with BlogAddOn installed without the mentioned error of not-found installer in the working directory.

Summary: I hope this provides sufficient detail to pin down the culprit.

-- AndriusJuodagalvis - 07 Oct 2010

---

I'll run through your steps again. One thing I'm missing is where in the Configure UI those errors show up. Each field in configure can have it's own checker, implemented as a separate module. So it would be very helpful to know where the errors are presented. (Which major tab, minor tab and field).

As far as #2, BlogAddOn is not a Plugin, so it won't show up in the list of InstalledPlugins. However as you noted, it does show up under the list of "All Contrib Modules".

For #3, the script (and Web based installation) should save a copy of the _installer module into working/configure/pkgdata, the manual un-tar does not. Although this sounds backwards, because you get the not-found message when you run the script.

-- GeorgeClark - 07 Oct 2010

I'm sorry to say, that for #1, I believe I've followed your exact same steps:

• Untar'd a fresh install of Foswiki 1.1 into a new directory - /var/www/foswiki11/
• chown -R apache:apache foswiki11
• ran configure, and saved the guessed path settings, and again to fix the email id and server.
  • One warning about missing .htpasswd file remains - ignored.
• Downloaded CommentPlugin_installer into web root and ran it answering yes to all questions

Do you want to use locally found installer scripts and archives to install CommentPlugin and any dependencies.
If you reply n, then fresh copies will be downloaded from this repository.? [y/n] y
Package repository set to http://foswiki.org/pub/Extensions 

 ... locally found installer scripts and archives will be used if available in /var/www/foswiki110
Using local /var/www/foswiki110/CommentPlugin_installer for package manifest 

...  (omitted boilerplate)

CommentPlugin version $Rev: 8843 (2010-09-01) $ is already installed. Are you sure you want to re-install this module? [y/n] y

CommentPlugin ready to be installed Do you want to proceed with installation of CommentPlugin? [y/n] y
===== INSTALLING CommentPlugin 
Creating Backup of CommentPlugin ...
Backup saved into /var/www/foswiki110/working/configure/backup/CommentPlugin-backup-20101007-111226 
   Archived as /var/www/foswiki110/working/configure/backup/CommentPlugin-backup-20101007-111226.tgz 
Running Pre-install exit for CommentPlugin ...
Installing CommentPlugin... 
No local package found, and uselocal requested - download required
Unpacking /tmp/4lCNKYl8qX.tgz...
Checked in: data/Sandbox/CommentPluginExampleComments.txt  as Sandbox.CommentPluginExampleComments
Checked in: data/Sandbox/CommentPluginExamples.txt  as Sandbox.CommentPluginExamples
Checked in: data/Sandbox/CommentPluginTemplateExample.txt  as Sandbox.CommentPluginTemplateExample
Checked in: data/System/CommentPlugin.txt  as System.CommentPlugin
Attached:   pub/System/CommentPlugin/wikiringlogo20x20.png to System/CommentPlugin
Checked in: data/System/CommentPluginTemplate.txt  as System.CommentPluginTemplate
Checked in: data/System/VarCOMMENT.txt  as System.VarCOMMENT
Installed:  lib/Foswiki/Plugins/CommentPlugin.pm
Installed:  lib/Foswiki/Plugins/CommentPlugin/Comment.pm
Installed:  templates/comments.tmpl
Installed:  CommentPlugin_installer
Running Post-install exit for CommentPlugin...

Note: Don't forget to enable installed plugins in the
"Plugins" section of bin/configure, listed below:

  CommentPlugin 

No errors encountered during installation

• chown -R apache:apache *
• re-loaded configure. no errors

So at this point I'm wondering if there is something unique about ScientificLinux5 What version of perl do you have installed?

-- GeorgeClark - 07 Oct 2010

One more thought - could you also look at the permissions and ownership of the Directories that contain the files triggering the errors: /var/www/foswiki/Foswiki-1.1.0-20101007/templates and /var/www/foswiki/Foswiki-1.1.0-20101007/working/configure/pkgdata

-- GeorgeClark - 07 Oct 2010

---

I repeated the steps and still get the error message. The requested info and remarks:

1. perl version is 5.8.8 for x86_64-linux-thread-multi
2. The output from CommentPlugin_installer is identical to the one you have presented with the differences in the names of temporary files. Thus this step is ok.
3. ownership of templates, working/configure/pkgdata is apache:apache. Permissions

   drwxr-xr-x  2 apache apache   4096 Oct  8 11:36 templates

   The same for the other directory.
4. The configurator, when run from a web browser, reports errors in those places:
   • Tab "General path settings"
     • Field {TemplateDir} has the value /var/www/foswiki/Foswiki-1.1.0-20101007/template
       • below there is a statement

         Error: /var/www/foswiki/Foswiki-1.1.0-20101007/templates/comments.tmpl cannot be found

     • Field {WorkingDir} ... (similarly reports on CommentPlugin_installer)
   • Tab "Extensions"
     • Field "EnabledPlugins"
       • Message below the subfield "{PluginsOrder}" :

          Error: Foswiki::Plugins::CommentPlugin is enabled in LocalSite.cfg but was not found in the path

5. Now the suspicion fell on the configure script itself (or its subscripts): maybe it fails, if it verifies only the updated items. (It would be nice, if it would save the working log somewhere.) I decided to run the configuration script interactively `perl -T ./configure`. It reported:

   <h1>Software error:</h1>
   <pre>Use of uninitialized value in pattern match (m//) at /var/www/foswiki/Foswiki-1.1.0-20101007/lib/Foswiki/Configure/Checkers/ScriptSuffix.pm line 14.
   </pre>
   

6. I continued to pursue the idea of verifying that the script for some reason checks the timestamp of the file. Surprisingly,

   touch --reference=compare.tmpl comments.tmpl

   did the trick of removing an error for this file. The same procedure did not work of the CommentPlugin_installer in the working area. Thus this file is not found and the Comment plugin is disabled.
7. I could not reproduce the trick with the time stamp. However, the side observation could be repeated. If I rename the comment.tmpl file to say comment.tmp2l. The configure reports comment.tmp2l as not found. (Ownerships were set apache:apache.)

-- AndriusJuodagalvis - 08 Oct 2010

---

Looking through the prerequisities for foswiki, I recalled that I have no CPAN module Config (the required version is >=0). In fact, perl says that the Config is up to date (undef).

-- AndriusJuodagalvis - 08 Oct 2010

I am really baffled about this. The TemplateDir error - the code checks all files to see that they are readable. The routine is lib/Foswiki/Configure/Checkers/TemplateDir.pm and the line that generates the not-found message is in lib/Foswiki/Configure/Checker.pm in sub checkTreePerms

    return $path . ' cannot be found' . CGI::br()
      unless ( -e $path || -l $path );

• -e - File or directory name exists
• -l - Entry is a symbolic link

So if the file exists, or is a link, then the message should not be generated. So something strange is going on with perl from what I can see. This just should not be failing.

The PluginsOrder checker is doing the same test. It's a bit more complicated - looking for the plugin module along the various paths where it might be found, but it's the same -e test: if ( -e "$dir/$enabled/$plugpath.pm" ) {

If the "-e" test is failing, then something really strange is going on. You've pretty much proven that the files are really there.

As far as running configure from the shell, the uninitialized variable is an ENV hash that is set by Apache. Configure isn't fully functional without being in a CGI environment.

I talked about this issue on IRC a bit with pharvey. A couple of more questions - are:

• What file system are you using. It's suspicious that a "touch" would resolve the issue
• Are you using SELinux or other enhanced security that might be coming into play?

If you have the time, you might come up on the #foswiki IRC channel to see if any of the dev's are around to dig into this a bit further interactively.

-- GeorgeClark - 09 Oct 2010

---

A few answers:

• File system is ext3
• SELinux is permissive/disabled (I tried both) as a process, yet httpd loads it.

  [Sun Oct 10 11:03:12 2010] [notice] SELinux policy enabled; httpd running as con
  text user_u:system_r:httpd_t:s0
  

What puzzles me is the persistence of the "problematic file." I will explain this in a minute. I rename the "unfound" file from comments.tmpl to comments2.tmpl. Then the file comments2.tmpl is "not found." I copy the file comments2.tmpl to comments.tmpl. The file comments.tmpl is not reported as unfound, yet comments2.tmpl is not found. I could further rename the file to comments3.tmpl, and this file will not be found. I could make the configure to forget the "missing" files by renaming the "problematic" files and copying them back to the expected name. At the end, configure "agreed" that the CommentPlugin is installed and could be enabled. Foswiki showed it as an enabled plugin. (The files comments2.tmpl and CommentPlugin _installer2 are still reported as not found. The files in lib/.../CommentPlugin* had to be renamed and copied back.)

Another endeavour. By setting the variable SCRIPT_NAME, I was able to run the configure script from a command line. The created html file contained no messages about the unfound files, and CommentPlugin could be enabled according to its decision (it was listed among the enabled plugins), yet the foswiki system was showing it as a disabled plugin.

So, the problem could be in the configuration of SELinux loaded by the web server.

-- AndriusJuodagalvis - 10 Oct 2010

Are you by any chance running configure under a persistent perl - loading it into memory with mod_perl, fastcgi, fcgid, or the old speedy cgi? Configure is not written to be kept memory resident and reused, so that might explain the strange persistence of the "missing" file. The config checker reads the directory fresh every time, a file change should really be detected.

-- GeorgeClark - 10 Oct 2010

I guess I did not convey the message, although mod_perl is indeed loaded (I did not reported that I actually reloaded configure page each time after the changes, instead of simply moving back and forth between the tabs). The persistence that I wrote about is something that one would expect from a software like selinux. The physical files seem to be tagged and their change (like rename) is followed up, yet their copying gets unnoticed. The config checker reported the renamed file as being missing, however, the "missing" file were ok after they were copied back (thus, there were two identical files: the "faulty" file, which is reported as "not found" by configure, and its copy, which is not reported as "not found"). As I wrote, moving and copying files forced the plugin to be ok. Thus this would be a work-around.

I am only wondering whether having configure in the web browser could make the security system to tag the (physical) files. I might verify this later.

-- AndriusJuodagalvis - 11 Oct 2010

---

Finally, things got separated into boxes. Thanks George, for taking the discussion.

Once I discovered the initial problem, I scanned foswiki.org for a possible solution and found it in a similar question-answers discussion (while I think the solution should be put into Foswiki support FAQ or best practice tips). There someone remarked that there is a simple aid for problems with selinux, setroubleshoot. I installed the package, however, in the log files at /var/log/setroubleshoot there were no reports that would explain the difficulties that I was experiencing (in fact, these logs are empty now). Thus I decided to ask a question on the foswiki support page.

Today, having decided to drop the pursuit, I still checked the documentation of the setroubleshoot, since the problem seemed to be stemming from the selinux package. There was an explanation that the problem database is located at /var/lib/setroubleshoot/database.xml, the logs only report problems of the setroubleshoot itself. Moreover, a user can start a program sealert which would warn that selinux took some action. I checked the database, and there it was - an answer to the observed problem. The troubleshooter in plain language explained that selinux was hiding the files from bin/configure as it thought the files were set in a wrong way. The file also contained the instructions of how to solve the issue (if the selinux assumptions were incorrect). After I followed them, bin/configure reported no more problems.

I would recommend that the setroubleshoot would be advertised on Foswiki support FAQ, as the suggestions to disable selinux are abound while this is a nice project.

Thanks George for the support, and the developers' team for working on foswiki.

-- AndriusJuodagalvis - 12 Oct 2010