Item13845: bulk_copy.pl fails to copy WebPreferences topic if RcsWrap is in use.
Priority: Urgent
Current State: Closed
Released In: 2.0.3
Target Release: patch
User reported on IRC
Copy topic Main/ComputationalPhysics.WebPreferences
Version 1 by UserRetacted at 09 Jan 2014 - 04:04
$VAR1 = \bless( {
'-file' => '/var/www/Foswiki-2.0.2/lib/Foswiki/Store/Rcs/RcsWrapHandler.pm',
'-text' => 'ci -m%COMMENT|U% -t-none -d%DATE|D% -u -w%USERNAME|S% %FILENAME|F% of .../ComputationalPhysics/WebPreferences.txt failed: 1 ',
'-line' => 156,
'-package' => 'Foswiki::Store::Rcs::RcsWrapHandler'
}, 'Error::Simple' );
Uncaught exception from user code:
Don't know how to convert a Error::Simple at bulk_copy.pl line 109.
main::convert('Error::Simple=HASH(0x38ad9e0)', 'to_unicode') called at bulk_copy.pl line 440
main::dispatch('["saveTopicRev","Main/ComputationalPhysics","WebPreferences",...') called at bulk_copy.pl line 675
main::convert('Error::Simple=HASH(0x38ad9e0)', 'to_unicode') called at bulk_copy.pl line 440
main::dispatch('["saveTopicRev","Main/ComputationalPhysics","WebPreferences",...') called at bulk_copy.pl line 675
Bad response from peer at bulk_copy.pl line 152, <FROM_B> line 1058.
This is after the following patch was applied to get more information:
diff --git a/core/tools/bulk_copy.pl b/core/tools/bulk_copy.pl
index 17bb4cf..9f2bdea 100755
--- a/core/tools/bulk_copy.pl
+++ b/core/tools/bulk_copy.pl
@@ -104,6 +104,8 @@ sub convert {
elsif ( ref($item) ) {
# CODE, REF, GLOB, LVALUE, FORMAT, IO, VSTRING, Regexp
+ use Data::Dumper;
+ print STDERR Data::Dumper::Dumper( \$item );
Carp::confess "Don't know how to convert a " . ref($item);
}
elsif ($Foswiki::UNICODE) {
--
GeorgeClark - 10 Nov 2015
They report that it's failing on every
WebPreferences topic. Crawford, any ideas?
Marking this urgent, the user reported that the webs are all operational but possibly with a skeleton
WebPreferences.txt, which means they lost all ACLs for the copied webs.
--
GeorgeClark - 10 Nov 2015
Recreated. Works fine with
RcsLite. Fails as above with
RcsWrap. rcs is failing to do the checkin. The problem is related to how populateWeb creates a new Web. It starts by creating a new WebPreferences topic using today's date, and then saves it with the templated changes, again using today's date. Then later when
bulk_copy
copies the topics, it tries to copy WebPreferences rev 1 from the source web over the top of the newly created web. RcsLite is happy with this, but RcsWrap fails, rejecting the "backdating" of rev 1.
--
GeorgeClark - 10 Nov 2015
I have a patch, but it tinkers with the Meta API. CDot, if there is any chance you could review. I'd like to get this in to 2.0.3 .. I'm building it on Friday 13 Nov. If I don't hear, my alternative is to make
bulk_copy
die if RcsWrap is in use.
diff --git a/core/lib/Foswiki/Meta.pm b/core/lib/Foswiki/Meta.pm
index 416b371..973b3ce 100644
--- a/core/lib/Foswiki/Meta.pm
+++ b/core/lib/Foswiki/Meta.pm
@@ -819,7 +819,13 @@ with 'Web' will be copied. If no base web is specified, an empty web
an error will be thrown.
$opts is a ref to a hash that contains settings to be modified in
-the web preferences topic in the new web.
+the web preferences topic in the new web, and also allows the web
+create date to be overridden.
+
+If $opts->{webdate} is set to a valid date, the WebPreferences topic
+will be created with the specified date as its creation date.
+This option is used by the bulk_copy utility to accurately copy the
+revisions of the WebPreferences topic.
=cut
@@ -858,6 +864,7 @@ sub populateNewWeb {
# Make sure there is a preferences topic; this is how we know it's a web
my $prefsTopicObject;
+ my $webPrefsDate;
if (
!$session->topicExists(
$this->{_web}, $Foswiki::cfg{WebPrefsTopicName}
@@ -865,10 +872,13 @@ sub populateNewWeb {
)
{
my $prefsText = 'Preferences';
+ $webPrefsDate = $opts->{webdate};
+ delete $opts->{webdate};
+
$prefsTopicObject =
$this->new( $this->{_session}, $this->{_web},
$Foswiki::cfg{WebPrefsTopicName}, $prefsText );
- $prefsTopicObject->save();
+ $prefsTopicObject->save( forcedate => $webPrefsDate, );
}
if ($templateWeb) {
@@ -956,7 +966,7 @@ s/($Foswiki::regex{setRegex}$key\s*=).*?$/$1 $opts->{$key}/gm
}
}
$prefsTopicObject->text($text);
- $prefsTopicObject->save();
+ $prefsTopicObject->save( forcedate => $webPrefsDate, );
}
}
diff --git a/core/tools/bulk_copy.pl b/core/tools/bulk_copy.pl
index 17bb4cf..31e30de 100755
--- a/core/tools/bulk_copy.pl
+++ b/core/tools/bulk_copy.pl
@@ -216,9 +216,15 @@ sub copy_web {
announce "Copy web $web";
my $webMO = Foswiki::Meta->new( $session, $web );
+ my $webPT =
+ Foswiki::Meta->new( $session, $web, $Foswiki::cfg{WebPrefsTopicName} );
+ $webPT->load(1);
+ my $webPTinfo = $webPT->get('TOPICINFO'); # get the TOPICINFO hash
+ my $webPTdate = $webPTinfo->{date};
+
# saveWeb returns the name of the WebPreferences topic,
# iff it was created
- my $wpt = call( 'saveWeb', $web );
+ my $wpt = call( 'saveWeb', $web, $webPTdate );
my $tit = $webMO->eachTopic();
while ( $tit->hasNext() ) {
my $topic = $tit->next();
@@ -465,11 +471,13 @@ sub topicExists {
# the name of the WebPreferences topic (this is used by some stores to
# identify a web)
sub saveWeb {
- my $web = shift;
+ my $web = shift;
+ my $webDate = shift;
+
return '' if $session->webExists($web);
my $mo = Foswiki::Meta->new( $session, $web );
return '' if $control{check_only};
- $mo->populateNewWeb();
+ $mo->populateNewWeb( undef, { webdate => $webDate } );
# Return the name of the web preferences topic, as it was just created
return $Foswiki::cfg{WebPrefsTopicName};
@@ -480,7 +488,7 @@ sub saveWeb {
sub saveTopicRev {
my ( $web, $topic, $data ) = @_;
- my $mo = Foswiki::Meta->new( $session, $web, $topic );
+ my $mo = Foswiki::Meta->new( $session, $web, $topic, { forcedate => 1 } );
$mo->setEmbeddedStoreForm($data);
my $info = $mo->get('TOPICINFO');
--
GeorgeClark - 10 Nov 2015
And this change breaks
StoreTests, and I have no idea why. I can't check it in.
--
GeorgeClark - 11 Nov 2015
Following patch works. Checking it in for 2.0.3. Hopefully you'll have a better fix.
diff --git a/core/tools/bulk_copy.pl b/core/tools/bulk_copy.pl
index 17bb4cf..9a40b01 100755
--- a/core/tools/bulk_copy.pl
+++ b/core/tools/bulk_copy.pl
@@ -663,6 +663,11 @@ else {
binmode( STDOUT, ':utf8' ) if $Foswiki::UNICODE;
+ if ($Foswiki::cfg{Store}{Implementation} =~ m/RcsWrap$/ ) {
+ announce "Overriding target Store Implementaion to 'RcsLite'";
+ $Foswiki::cfg{Store}{Implementation} = 'Foswiki::Store::RcsLite';
+ }
+
announce
"Copying to $Foswiki::VERSION ($Foswiki::cfg{Store}{Implementation})";
--
GeorgeClark - 11 Nov 2015
Hi George, so basically for now is to have the script automatically do what you told me to do in the IRC? (Use
RcsLite instead of
RcsWrap) Is there a performance hit by using
RcsLite? From my understanding
RcsLite uses a Perl library while
RcsWrap uses an external library so I was just wondering if there was a performance difference between the two.
--
AntelmoAguilar - 11 Nov 2015
There are subtle differences between the two.
RcsWrap "forks" to an external process. So if you are using
FastCGI or
especially mod_perl, then you want to avoid fork if at all possible. It's very costly. In that case
RcsLite is generally preferred. However
RcsLite reads in the entire change history into memory, even for minor operations which would only require the headers. So on topics with very large histories, or histories of large binary files,
RcsWrap is preferred.
The Plain File store attempts to avoid these conflicting issues, but is new, so has less operational experience.
--
GeorgeClark - 11 Nov 2015
From your description I'd say the problem here is with RCS, and not any part of Foswiki. Did you consider looking at the command-line options of RCS to see if there is one that allows backdating?
--
Main.CrawfordCurrie - 20 Nov 2015 - 09:27
I couldn't find any options that apply. I did dig into that as a possible fix.
--
GeorgeClark - 20 Nov 2015