You are here: Foswiki>Tasks Web>Item12084 (19 Nov 2013, GeorgeClark)Edit Attach

Item12084: Can't use an undefined value as a symbol reference at lib/Foswiki/Request/Upload.pm line 86.

pencil
Priority: Urgent
Current State: Closed
Released In: 1.1.9
Target Release: patch
Applies To: Engine
Component: CGI
Branches: Release01x01 trunk
Reported By: PhilCrooker
Waiting For:
Last Change By: GeorgeClark
I'm getting exactly the symptom of Item9418 - don't know if this is actually the same, but looks it. I'm running foswiki on opensuse 12.1.

When attempting to upload a file the web server returns:

Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.

Can't use an undefined value as a symbol reference

In the error.log:

| 2012-09-22T10:40:33Z warning | Can't use an undefined value as a symbol reference at /srv/www/htdocs/foswiki/lib/Foswiki/Request/Upload.pm line 86. at /srv/www/htdocs/foswiki/lib/Foswiki/Request/Upload.pm line 86 Foswiki::Request::Upload::handle('Foswiki::Request::Upload=HASH(0x11cbdc0)') called at /srv/www/htdocs/foswiki/lib/Foswiki/Request.pm line 636 Foswiki::Request::upload('Foswiki::Request=HASH(0x10ee890)', 'filepath') called at /srv/www/htdocs/foswiki/lib/Foswiki/UI/Upload.pm line 161 Foswiki::UI::Upload::_upload('Foswiki=HASH(0x1114fa0)') called at /srv/www/htdocs/foswiki/lib/Foswiki/UI/Upload.pm line 87 Foswiki::UI::Upload::upload('Foswiki=HASH(0x1114fa0)') called at /srv/www/htdocs/foswiki/lib/Foswiki/UI.pm line 316 Foswiki::UI::__ANON__() called at /srv/www/htdocs/foswiki/lib/CPAN/lib/Error.pm line 379 eval {...} called at /srv/www/htdocs/foswiki/lib/CPAN/lib/Error.pm line 371 Error::subs::try('CODE(0x7865c8)', 'HASH(0x1114b68)') called at /srv/www/htdocs/foswiki/lib/Foswiki/UI.pm line 435 Foswiki::UI::_execute('Foswiki::Request=HASH(0x10ee890)', 'CODE(0x786610)', 'upload', 1) called at /srv/www/htdocs/foswiki/lib/Foswiki/UI.pm line 274 Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x10ee890)') called at /srv/www/htdocs/foswiki/lib/Foswiki/Engine/CGI.pm line 41 Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0xa5da08)') called

-- PhilCrooker - 22 Sep 2012

  • Which version of perl are you using perl -v
  • Are Locales enabled? Setting in LocalSite.cfg is $Foswiki::cfg{UseLocale}
  • Is a character set specified? $Foswiki::cfg{Site}{CharSet}
  • Which CGI version? (See bin/configure in the Web Server Environment tab, the Perl Modules section, will state something like "3.51 installed. for base Foswiki ..."

-- GeorgeClark - 22 Sep 2012

Thanks for the help. Responses:

  • perl v 5.14.2
  • UseLocale was set to 0, changed it to 1, still get error.
  • character set is: 'iso-8859-1'
  • CGI version 3.52

-- PhilCrooker - 23 Sep 2012

We ran into another user with this issue, and with some debugging on IRC, got to the cause at least for their situation.

  • $Foswiki::cfg{Register}{AllowLoginName} = 1; (Allow login names was enabled)
  • The Main/WikiUsers topic was corrupted. (Blank line ahead of the META:TOPICINFO line)

If you are using the TopicUserMapping contrib, make sure that the /data/System/WikiUsers.txt file is a valid topic - especially no extra blank lines in the list itself, and a valid TOPICINFO as the very first line.

-- GeorgeClark - 04 Oct 2012

Yes I also have login names enabled. I had played around with the initial install - manually editing files - and probably screwed something up. I then re-installed and it worked OK. I then ran the Nikto scanner against the system and that broke it again! Restoring to pre-scan files fixed it. I tried substituting a few files from the broken version but couldn't get it to fail again. Anyway, now working and will keep those backups happening....

-- PhilCrooker - 18 Oct 2012

This happened again, I manually edited WikiUsers using vi, just removing a user, and it broke again as above. Restoring the WikiUsers file fixed it. If I remove the user via the wiki edit it works fine.

According to the CurrentState page I'm supposed to "flip" this task to new which I just did. Should the task just be closed? Your call.

-- PhilCrooker - 28 Oct 2012

No, there is something wrong if this type of failure can occur. We should be a bit more robust in handling the WikiUsers topic.

The only way I can recreate it is by inserting a line in front of the %META:TOPICINFO line, which must be the first line of the file. It seems to cause the temporary file opened for the upload by CGI to be prematurely closed and released. So far I cannot figure out why. The mapping topic seems to be processed correctly, and the Login to wikiname mapping is valid.

When you edit your WikiUsers topic, what exactly do you do beyond deleting the line for the user. I can delete users, even the user currently being used to test the upload, and everything continues to work. It's only the extra line at the start that breaks things.

-- GeorgeClark - 28 Oct 2012

A bit more related information. It's also dependent upon RcsWrap. RcsLite doesn't fail. I believe it's something to do with Store or Meta triggering CGI to close it's open temp files. Maybe an internal error that's thrown and caught, so that it's masked. But so far I just can't find it.

-- GeorgeClark - 28 Oct 2012

I've had problems getting the error to happen consistently. I tried to upload an attachment and in the description I had "Anne and Barry's plan". This caused the error. I took out the " ' " and I could add that attachment. I tried again later with the " ' " in the description and no error.

Using vi, I previously tried just deleting a user entry in WikiUsers.txt (not touching the .txt,v file) along with that user entry in .htpasswd - sometimes this would trigger the error and other times not. Today, I again tried editing the file with vi and just deleting the user entry from WikiUsers.txt and .htpasswd I am consistently getting the error. I copy the unedited files back and the file upload works. I have been able to repeat this. This is all I do, just edit the files with vi and then as a normal user try to upload a file with a description - the user can login afterwards or be logged in during the editing, or restart apache, no matter. Over to you....

-- PhilCrooker - 25 Nov 2012

I just saw this again on a client site; vanilla 1.1.8 install, data came from an ancient TWiki. Unfortunately because it's a client site I can't debug it, but elevating to 'Urgent' cos it's not good. I can't see for the life of me why it's doing what it's doing, but it does seem to be blowing away the temp files attached to the uploads, and it can be reproduced with the client's working data.

A the very least there needs to be a description of the symptoms, and the workaround (switch to RcsLite) in the release notes.

Note: AllowLoginName is enabled, but the WikiUsers topic is not corrupt.

-- CrawfordCurrie - 17 Oct 2013

More information from the client:

But it might be worth mentioning that I accidentally mv'ed the WikiUsers (and also the WikiGroups) topic from the Main web when I was "cleaning things up" last Friday. I had spent most of the day using the newly set-up web and making sure foswiki was mostly working. Once I thought it was ok, I mv'ed topics out of Main to make extra sure that users wouldn't wind up editing them somehow. And in doing that I accidentally mv'ed a few that I should not have.....

Anyway: its possible that the WikiUsers topic files were corrupted somehow (date changed?) in this process.

I didn't know this before I researched it. I don't recall any problems with permissions or ownership of WikiUsers, though that looks like a likely cause.

-- CrawfordCurrie - 22 Oct 2013

I can't reproduce this. All we can do is wait for someone to report it again and then try and debug it in the system where it occurs. Since it can't be reproduced, I have dropped the priority to Normal again.

-- CrawfordCurrie - 23 Oct 2013

I have been able to reproduce this now, so have bumped it back to Urgent. The CGI temporary upload file is destroyed (!) on the first call to Foswiki::Sandbox::sysCommand, specifically when real pipes are used and the $data = <$handle> happens, the uploaded file simply disappears.

The problem manifests as described above because when AllowLoginName is enabled, the user mapping has to be loaded from the WikiUsers topic, which requires a readTopic call which, when RcsWrap is in use, results in a call to sysCommand.

I am working to narrow this down, to try and understand why this specific call to sysCommand causes this problem.

Later: It appears to be a bug in CGI. Here's the code arghno.pm used to reproduce it:
#!/usr/bin/perl
use strict;
use CGI;
use CGI::Carp ('fatalsToBrowser');

my $q = new CGI;
my $fn = $q->param('filepath');
my $f = $q->tmpFileName($fn);
my $handle;
my $pid = open( $handle, '-|' );
if ($pid) { # Parent process
    die "Tmpfile has gone!" unless -e $f;
    my $data = <$handle>;
    die "Tmpfile has gone!" unless -e $f;
}
# If you get here, it worked
And the HTML to invoke it:
<html>
  <body>
    <form enctype="multipart/form-data" action="/foswiki/Release01x01/core/bin/arghno" method="post">
      <input type="file" name="filepath" value="" />
      <input type="submit" value='Upload file' />
    </form>
  </body>
</html>
The problem is the enctype on the form. CGI handles mutlipart forms by caching the form parts in temporary files, and maintaining a record of those files. When the CGI object is DESTROYed, then the temp files are unlinked. However, when a real pipe is created to run the sandbox command, the process is forked. When the child process exits - which it does fairly quickly - then the inherited CGI object in the child process is destroyed. This destroys the temp files.

Later still: This patch fixes it by disabling the CGITempFiles::DESTROY method in sub-processes:
Index: Foswiki/Engine/CGI.pm
===================================================================
--- Foswiki/Engine/CGI.pm   (revision 16728)
+++ Foswiki/Engine/CGI.pm   (working copy)
@@ -23,6 +23,38 @@
 use Foswiki::Request::Upload ();
 use Foswiki::Response        ();
 
+# ****
+# CGI.pm has a private class called CGITempFile which is used to hold
+# a handle to a temporary file. The idea is that when the CGI object
+# is reaped, the temporary files are automatically unlinked. However,
+# when Sandbox::sysCommand is invoked - for example, by loading a topic
+# before the uploads have been processed - a subprocess is created that
+# gets a copy of the parent process, including these objects. When the
+# subprocess terminates, the CGITempFiles are DESTROYed and the external
+# files unlinked, leaving the parent process with pointers to non-
+# existant temporary files.
+#
+# We solve this problem by patching the DESTROY method on CGITempFile so
+# that the temporary files are only ever destroyed by the parent process,
+# and never by the child. We do this by comparing the (inherited) value of
+# $$ with a value recorded when the CGI object is constructed.
+
+our $MASTER_PROCESS;
+our $SAVE_DESTROY;
+eval {
+    # In an eval in case it's ever removed from CGI
+    $SAVE_DESTROY = \&CGITempFile::DESTROY;
+};
+if ( defined $SAVE_DESTROY ) {
+    no warnings 'redefine';
+    *CGITempFile::DESTROY = sub {
+   if ( $$ == $MASTER_PROCESS ) {
+       # Parent process, unlink the temp file
+       &$SAVE_DESTROY(@_);
+   }
+    };
+}
+
 sub run {
     my $this = shift;
     unless ( $Foswiki::cfg{isVALID} ) {
@@ -146,6 +178,11 @@
     my ( $this, $req ) = @_;
 
     return unless $ENV{CONTENT_LENGTH};
+
+    # Record the master process so we don't reap temp files in
+    # sub-processes (see long comment ****)
+    $MASTER_PROCESS = $$;
+
     my $cgi = new CGI();
     my $err = $cgi->cgi_error;
     throw Foswiki::EngineException( $1, $2 )

Reported upstream as https://rt.cpan.org/Ticket/Display.html?id=89747

-- CrawfordCurrie - 24 Oct 2013

I'm seeing errors in the Unit Tests:

Running UploadScriptTests
        UploadScriptTests::test_linkformat
        (in cleanup) Use of uninitialized value $Foswiki::Engine::CGI::CONSTRUCTOR_PID in numeric eq (==) at /var/www/foswiki/trunk/core/lib/Foswiki/Engine/CGI.pm line 51.
        (in cleanup) Use of uninitialized value $Foswiki::Engine::CGI::CONSTRUCTOR_PID in numeric eq (==) at /var/www/foswiki/trunk/core/lib/Foswiki/Engine/CGI.pm line 51.
        (in cleanup) Use of uninitialized value $Foswiki::Engine::CGI::CONSTRUCTOR_PID in numeric eq (==) at /var/www/foswiki/trunk/core/lib/Foswiki/Engine/CGI.pm line 51.
        UploadScriptTests::test_imagelink
        (in cleanup) Use of uninitialized value $Foswiki::Engine::CGI::CONSTRUCTOR_PID in numeric eq (==) at /var/www/foswiki/trunk/core/lib/Foswiki/Engine/CGI.pm line 51.
        (in cleanup) Use of uninitialized value $Foswiki::Engine::CGI::CONSTRUCTOR_PID in numeric eq (==) at /var/www/foswiki/trunk/core/lib/Foswiki/Engine/CGI.pm line 51.

Changing the patch eliminates the errors:
        if ( defined $CONSTRUCTOR_PID && $$ == $CONSTRUCTOR_PID ) {

Checking in a fix, needs Crawford's review.

-- GeorgeClark - 27 Oct 2013

Yes, that should be fine. The error is in the unit tests, not the code, but the fix is fine too.

-- CrawfordCurrie - 28 Oct 2013
 

ItemTemplate edit

Summary Can't use an undefined value as a symbol reference at lib/Foswiki/Request/Upload.pm line 86.
ReportedBy PhilCrooker
Codebase 1.1.8, 1.1.5
SVN Range
AppliesTo Engine
Component CGI
Priority Urgent
CurrentState Closed
WaitingFor
Checkins distro:95c4d7f72fca distro:b8a8c318f34f distro:73e2124bf2e2 distro:9f97965ce6e7
TargetRelease patch
ReleasedIn 1.1.9
CheckinsOnBranches Release01x01 trunk
trunkCheckins distro:b8a8c318f34f distro:73e2124bf2e2
Release01x01Checkins distro:95c4d7f72fca distro:9f97965ce6e7
Topic revision: r26 - 19 Nov 2013, GeorgeClark - This page was cached on 31 May 2016 - 10:36.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License