Item13313: Issues encountered converting RCS store to PlainFile

Ran into several issues.

1. Pub directory skipped large number of files. pub/System/FamFamFamSilkIcons/ 1003 files, only 1 file in converted directory.
2. One of my topics ended up as a back rev. I think it had been manually edited, but it seems to have been restored to the last RCS version instead of the cached text file.
3. It didn't copy any subwebs. I think the following is the fix:

diff --git a/PlainFileStoreContrib/tools/change_store.pl b/PlainFileStoreContrib/tools/change_store.pl
index b32b4cf..e2c8f64 100644
--- a/PlainFileStoreContrib/tools/change_store.pl
+++ b/PlainFileStoreContrib/tools/change_store.pl
@@ -212,7 +212,7 @@ switch_dirs(1);
 my $target_store = $target->new();
 
 switch_dirs(0);
-my $wit = $source_store->eachWeb('');
+my $wit = $source_store->eachWeb('', 1);
 while ( $wit->hasNext() ) {
     my $web_name = $wit->next();
     next unless $web_name =~ /^($weblist)$/o;

-- GeorgeClark - 20 Mar 2015

1. There is no META:FILEATTACHMENT for any of those files. They only "exist" by virtue of their presence in the pub directory, they have no history and are not part of the topic.
2. Need an example
3. Your patch is correct, thanks

-- CrawfordCurrie - 21 Mar 2015

1. I enabled AutoAttachPubFiles, and the missing files show up as attachments when the topic is viewed, but the meta is not updated. I tried adding AutoAttach to the settings used by change_store but the files are still ignored. So it appears that in the situation of any customer using AutoAttach, all of the pub files will be lost. So I think that this still needs to be fixed. yikes Just noticed that AutoAttachPubFiles has been silently removed from Foswiki.spec. granted it's "considered experimental" but was never announced as deprecated. I think we have to put that back. I know we have customers using this feature.
2. Here is an example.
3. Applied.

See Item13313.tgz: Topic reverted during store migration Contains the failing topic & rcs file:

gac@cardinal: /var/www/foswiki/distro/core (master)$ perl -I lib tools/change_store.pl -w Usersweb -i AdminUser -t ./data2 RcsLite PlainFile
Options:
-sd /var/www/foswiki/distro/core/data
-sp /var/www/foswiki/distro/core/pub
-td ./data2/data
-tp ./data2/pub
Scanning web Usersweb
... copy AdminUser:1
... copy AdminUser:2
... copy AdminUser:3
gac@cardinal: /var/www/foswiki/distro/core (master)$ vi data2/data/Usersweb/AdminUser.txt 
gac@cardinal: /var/www/foswiki/distro/core (master)$ vi data2/data/Usersweb/AdminUser.txt 
gac@cardinal: /var/www/foswiki/distro/core (master)$ tar -czvf Item13313.tgz data/Usersweb/AdminUser*
data/Usersweb/AdminUserLeftBar.txt
data/Usersweb/AdminUser.txt
data/Usersweb/AdminUser.txt,v
gac@cardinal: /var/www/foswiki/distro/core (master)$ diff data/Usersweb/AdminUser.txt data2/data/Usersweb/AdminUser.txt 
1c1
< %META:TOPICINFO{author="ProjectContributor" date="1300557820" format="1.1" version="1"}%
---
> %META:TOPICINFO{author="BaseUserMapping_333" date="1404009335" format="1.1" version="3"}%
36a37,44
> %META:PREFERENCE{name="SKIN" title="SKIN" type="Set" value="famfamfam,natedit,pattern"}%
> %META:PREFERENCE{name="FOSWIKI_LAYOUT_URL" title="FOSWIKI_LAYOUT_URL" type="Set" value="/pub/System/PatternSkinTheme2012/layout.css"}%
... 6 more lines of META differences

---

The Silk topics have all been fixed, under task Item13318

New instance, same issue. TinyMCEPlugin doesn't convert pub. None of the javascript is attached. I suspect that there are other System topics that store data in subdirectories that will be missed. I think these really need to be handled. We can't ship a product that is unable to convert itself.

/var/www/foswiki/distro/core (master)$ find pub/System/TinyMCEPlugin/ -type f | wc -l
735
/var/www/foswiki/distro/core (master)$ find data2/pub/System/TinyMCEPlugin/ -type f | wc -l
3

-- GeorgeClark - 21 Mar 2015

I don't understand the problem. If AutoAttachPubFiles is enabled, all it does is to generate a META:FILEATTACHMENT on the fly for files that appear in the pub directory. If AutoAttachPubFiles is not enabled, then those files do not exist as far as the store (Foswiki) is concerned.

It so happens that when AutoAttachPubFiles is disabled, files in the pub directory are still visible because pub is a public directory to the web server.

I'm not sure how AutoAttachPubFiles works, but if it's even vaguely sensible it will work in the store and make such files look like any other attachment. It is implemented entirely in the RCSStoreContrib, and change_store knows nothing about it, nor should it need to know.

BTW AutoAttachPubFiles was renamed to {RCS}{AutoAttachPubFiles} (It's specific to the RCSStoreContrib).

-- CrawfordCurrie - 22 Mar 2015

I think this is closed, but leaving it for George's feedback as he is the final arbiter of that.

-- CrawfordCurrie - 26 Mar 2015

Only thing would be maybe some docs to point out that subdirectories are not handled. Not sure if there is any chance of warning about them during the audit.

-- GeorgeClark - 27 Mar 2015

Crawford, I've attached a patch that allows convert_store to detect and warn if a pub subdir is detected. Didn't want to check it in as it does add a flag to the eachAttachment() API to request that subdirectory names be returned. I also added some "no warnings 'undefined'; to quiet things down a bit.

• patchit: Patch for detecting and warning about directory names.

-- GeorgeClark - 12 Apr 2015

That should be ok, but please document the hell out of it. As reluctant as I am to accept something like that into the public API, I guess it has to be done.

-- CrawfordCurrie - 12 Apr 2015

It probably doesn't have to be done, but its probably easier to do it this way. The other solution I can think of would be to do a separate scan of the store to look for and report subdirectories. I'll check it in.

-- GeorgeClark - 12 Apr 2015

Commenting is disabled while not logged in