How do I debug calls to third-party programs?

Debugging third-party programs

Foswiki uses a number of third-party programs that it invokes on the server, including rcs and grep. Sometimes you get an error from Foswiki that appears to come from one of these programs, usually when you try to save a topic. These errors usually look like crashes and result in mysterious messages in the browser, such as inappropriate ioctl for device or permission denied for exec or exec failed or even program failed.

It can be hard to work out what's going on here, because the third party programs often don't give much information to Foswiki. However you can use the following procedure to get more information.

First, find your Apache error log. The location of this file depends on OS/distribution and your local installation setup. Here are some typical locations:

• Debian: /var/log/apache2/error.log
• RHEL: /var/log/httpd/error_log
• Windows: accessible though the menus
• MacOS Leopard: /private/var/log/apache2/error_log

Look at the end of this file, and find the messages that relate to your error. You will probably see the error message you saw in the browser; for example:

[Mon Oct 19 10:16:01 2009] [error] [client 127.0.0.1] exec failed: Inappropriate ioctl for device
at /Library/WebServer/Documents/foswiki/lib/Foswiki/Sandbox.pm line 471., referer: http://127.0.0.1/foswiki/bin/view/Main/WebHome

Scan the log file back a bit; that might tell you immediately what's wrong. For example, if you had this message:

[Mon Oct 19 10:16:01 2009] configure: Can't exec "/usr/bin/rcs": No such file or directory

you would be able to tell that you don't have the "rcs" program installed, or the web server couldn't find it (perhaps because the path was wrong).

If you still can't work it out after reading the log file, try this.

1. Using a text editor, edit lib/Foswiki/Sandbox.pm in your Foswiki installation
2. Find the lines that say

   # Set to 1 to trace commands to STDERR
      sub TRACE { 0 }

3. Change the 0 to a 1.
4. Now run the command in Foswiki that gives the error.

If you look in your Apache error log file now, you will see the commands that Foswiki has used to call the external programs. For example,

[Thu Oct 22 07:48:17 2009] [error] [client 127.0.1.1] /usr/bin/rlog '-r1.1' '/var/www/foswiki/data/Sandbox/ControlledDocument.txt,v'
[Thu Oct 22 07:48:17 2009] [error] [client 127.0.1.1]  ->

In the example above, the command is /usr/bin/rlog '-r1.1' '/var/www/foswiki/data/Sandbox/ControlledDocument.txt,v', and the standard output of the command is shown after the arrow, -> (programs generate output in two ways; standard and error. The log only shows you the standard output, and this particular command doesn't generate any)

You can now copy-paste the command to a normal shell (terminal window, DOS box, whatever you call it), and observe the full error output there. Example:

bash$ /usr/bin/rlog '-h' '/home/foswiki/trunk/core/data/Sandbox/ControlledDocument.txt,v'
rlog: /home/foswiki/trunk/core/data/Sandbox/ControlledDocument.txt,v: Permission denied

Note that you have to run the command as a user with the same permissions as the web server user. This depends how you have set up your web server, but running the command as a root user (administrator) is usually the best approach.

Once you have corrected out the problem (it's often due to permissions, or broken softlinks, or missing programs, or wrong versions of programs, or similar) don't forget to turn the tracing back off again or the performance of your web server will suffer.

If you are still stuck (maybe you don't understand what the command-line run of the program is telling you) then you can contact other Foswiki users for help on IRC, or in the Support web.