You are here: Foswiki>Tasks Web>Item13845 (09 Dec 2015, GeorgeClark)Edit Attach

Item13845: bulk_copy.pl fails to copy WebPreferences topic if RcsWrap is in use.

pencil
Priority: Urgent
Current State: Closed
Released In: 2.0.3
Target Release: patch
Applies To: Engine
Component: FoswikiTools
Branches: master Item13699
Reported By: GeorgeClark
Waiting For:
Last Change By: GeorgeClark
User reported on IRC

Copy topic Main/ComputationalPhysics.WebPreferences
 Version 1 by UserRetacted at 09 Jan 2014 - 04:04
$VAR1 = \bless( {
                   '-file' => '/var/www/Foswiki-2.0.2/lib/Foswiki/Store/Rcs/RcsWrapHandler.pm',
                   '-text' => 'ci  -m%COMMENT|U% -t-none -d%DATE|D% -u -w%USERNAME|S% %FILENAME|F% of .../ComputationalPhysics/WebPreferences.txt failed: 1 ',
                   '-line' => 156,
                   '-package' => 'Foswiki::Store::Rcs::RcsWrapHandler'
                 }, 'Error::Simple' );
Uncaught exception from user code:
   Don't know how to convert a Error::Simple at bulk_copy.pl line 109.
      main::convert('Error::Simple=HASH(0x38ad9e0)', 'to_unicode') called at bulk_copy.pl line 440
      main::dispatch('["saveTopicRev","Main/ComputationalPhysics","WebPreferences",...') called at bulk_copy.pl line 675
   main::convert('Error::Simple=HASH(0x38ad9e0)', 'to_unicode') called at bulk_copy.pl line 440
   main::dispatch('["saveTopicRev","Main/ComputationalPhysics","WebPreferences",...') called at bulk_copy.pl line 675
Bad response from peer at bulk_copy.pl line 152, <FROM_B> line 1058.

This is after the following patch was applied to get more information:

diff --git a/core/tools/bulk_copy.pl b/core/tools/bulk_copy.pl
index 17bb4cf..9f2bdea 100755
--- a/core/tools/bulk_copy.pl
+++ b/core/tools/bulk_copy.pl
@@ -104,6 +104,8 @@ sub convert {
     elsif ( ref($item) ) {
 
         # CODE, REF, GLOB, LVALUE, FORMAT, IO, VSTRING, Regexp
+        use Data::Dumper;
+        print STDERR Data::Dumper::Dumper( \$item );
         Carp::confess "Don't know how to convert a " . ref($item);
     }
     elsif ($Foswiki::UNICODE) {

-- GeorgeClark - 10 Nov 2015

They report that it's failing on every WebPreferences topic. Crawford, any ideas?

Marking this urgent, the user reported that the webs are all operational but possibly with a skeleton WebPreferences.txt, which means they lost all ACLs for the copied webs.

-- GeorgeClark - 10 Nov 2015

Recreated. Works fine with RcsLite. Fails as above with RcsWrap. rcs is failing to do the checkin. The problem is related to how populateWeb creates a new Web. It starts by creating a new WebPreferences topic using today's date, and then saves it with the templated changes, again using today's date. Then later when bulk_copy copies the topics, it tries to copy WebPreferences rev 1 from the source web over the top of the newly created web. RcsLite is happy with this, but RcsWrap fails, rejecting the "backdating" of rev 1.

-- GeorgeClark - 10 Nov 2015

I have a patch, but it tinkers with the Meta API. CDot, if there is any chance you could review. I'd like to get this in to 2.0.3 .. I'm building it on Friday 13 Nov. If I don't hear, my alternative is to make bulk_copy die if RcsWrap is in use.
diff --git a/core/lib/Foswiki/Meta.pm b/core/lib/Foswiki/Meta.pm
index 416b371..973b3ce 100644
--- a/core/lib/Foswiki/Meta.pm
+++ b/core/lib/Foswiki/Meta.pm
@@ -819,7 +819,13 @@ with 'Web' will be copied. If no base web is specified, an empty web
 an error will be thrown.
 
 $opts is a ref to a hash that contains settings to be modified in
-the web preferences topic in the new web.
+the web preferences topic in the new web, and also allows the web
+create date to be overridden.
+
+If $opts->{webdate} is set to a valid date, the WebPreferences topic
+will be created with the specified date as its creation date.
+This option is used by the bulk_copy utility to accurately copy the
+revisions of the WebPreferences topic.
 
 =cut
 
@@ -858,6 +864,7 @@ sub populateNewWeb {
 
     # Make sure there is a preferences topic; this is how we know it's a web
     my $prefsTopicObject;
+    my $webPrefsDate;
     if (
         !$session->topicExists(
             $this->{_web}, $Foswiki::cfg{WebPrefsTopicName}
@@ -865,10 +872,13 @@ sub populateNewWeb {
       )
     {
         my $prefsText = 'Preferences';
+        $webPrefsDate = $opts->{webdate};
+        delete $opts->{webdate};
+
         $prefsTopicObject =
           $this->new( $this->{_session}, $this->{_web},
             $Foswiki::cfg{WebPrefsTopicName}, $prefsText );
-        $prefsTopicObject->save();
+        $prefsTopicObject->save( forcedate => $webPrefsDate, );
     }
 
     if ($templateWeb) {
@@ -956,7 +966,7 @@ s/($Foswiki::regex{setRegex}$key\s*=).*?$/$1 $opts->{$key}/gm
             }
         }
         $prefsTopicObject->text($text);
-        $prefsTopicObject->save();
+        $prefsTopicObject->save( forcedate => $webPrefsDate, );
     }
 }
 
diff --git a/core/tools/bulk_copy.pl b/core/tools/bulk_copy.pl
index 17bb4cf..31e30de 100755
--- a/core/tools/bulk_copy.pl
+++ b/core/tools/bulk_copy.pl
@@ -216,9 +216,15 @@ sub copy_web {
     announce "Copy web $web";
     my $webMO = Foswiki::Meta->new( $session, $web );
 
+    my $webPT =
+      Foswiki::Meta->new( $session, $web, $Foswiki::cfg{WebPrefsTopicName} );
+    $webPT->load(1);
+    my $webPTinfo = $webPT->get('TOPICINFO');    # get the TOPICINFO hash
+    my $webPTdate = $webPTinfo->{date};
+
     # saveWeb returns the name of the WebPreferences topic,
     # iff it was created
-    my $wpt = call( 'saveWeb', $web );
+    my $wpt = call( 'saveWeb', $web, $webPTdate );
     my $tit = $webMO->eachTopic();
     while ( $tit->hasNext() ) {
         my $topic = $tit->next();
@@ -465,11 +471,13 @@ sub topicExists {
 # the name of the WebPreferences topic (this is used by some stores to
 # identify a web)
 sub saveWeb {
-    my $web = shift;
+    my $web     = shift;
+    my $webDate = shift;
+
     return '' if $session->webExists($web);
     my $mo = Foswiki::Meta->new( $session, $web );
     return '' if $control{check_only};
-    $mo->populateNewWeb();
+    $mo->populateNewWeb( undef, { webdate => $webDate } );
 
     # Return the name of the web preferences topic, as it was just created
     return $Foswiki::cfg{WebPrefsTopicName};
@@ -480,7 +488,7 @@ sub saveWeb {
 sub saveTopicRev {
     my ( $web, $topic, $data ) = @_;
 
-    my $mo = Foswiki::Meta->new( $session, $web, $topic );
+    my $mo = Foswiki::Meta->new( $session, $web, $topic, { forcedate => 1 } );
     $mo->setEmbeddedStoreForm($data);
 
     my $info = $mo->get('TOPICINFO');

-- GeorgeClark - 10 Nov 2015

And this change breaks StoreTests, and I have no idea why. I can't check it in.

-- GeorgeClark - 11 Nov 2015

Following patch works. Checking it in for 2.0.3. Hopefully you'll have a better fix.
diff --git a/core/tools/bulk_copy.pl b/core/tools/bulk_copy.pl
index 17bb4cf..9a40b01 100755
--- a/core/tools/bulk_copy.pl
+++ b/core/tools/bulk_copy.pl
@@ -663,6 +663,11 @@ else {
 
     binmode( STDOUT, ':utf8' ) if $Foswiki::UNICODE;
 
+    if ($Foswiki::cfg{Store}{Implementation} =~ m/RcsWrap$/ ) {
+        announce "Overriding target Store Implementaion to 'RcsLite'";
+        $Foswiki::cfg{Store}{Implementation} = 'Foswiki::Store::RcsLite';
+    }
+
     announce
       "Copying to $Foswiki::VERSION ($Foswiki::cfg{Store}{Implementation})";
 

-- GeorgeClark - 11 Nov 2015

Hi George, so basically for now is to have the script automatically do what you told me to do in the IRC? (Use RcsLite instead of RcsWrap) Is there a performance hit by using RcsLite? From my understanding RcsLite uses a Perl library while RcsWrap uses an external library so I was just wondering if there was a performance difference between the two.

-- AntelmoAguilar - 11 Nov 2015

There are subtle differences between the two. RcsWrap "forks" to an external process. So if you are using FastCGI or especially mod_perl, then you want to avoid fork if at all possible. It's very costly. In that case RcsLite is generally preferred. However RcsLite reads in the entire change history into memory, even for minor operations which would only require the headers. So on topics with very large histories, or histories of large binary files, RcsWrap is preferred.

The Plain File store attempts to avoid these conflicting issues, but is new, so has less operational experience.

-- GeorgeClark - 11 Nov 2015

From your description I'd say the problem here is with RCS, and not any part of Foswiki. Did you consider looking at the command-line options of RCS to see if there is one that allows backdating?

-- Main.CrawfordCurrie - 20 Nov 2015 - 09:27

I couldn't find any options that apply. I did dig into that as a possible fix.

-- GeorgeClark - 20 Nov 2015
 
Topic revision: r9 - 09 Dec 2015, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy