Item12697: DirectedGraphPlugin use of Storable is platform dependent.

Priority: Normal
Current State: Closed
Released In: n/a
Target Release: n/a
Applies To: Extension
Component: DirectedGraphPlugin
Branches: trunk
Reported By: GeorgeClark
Waiting For:
Last Change By: GeorgeClark
Storable::freeze() and Storable::store() are platform dependent. If a Foswiki install is migrated from ia32 to x64, the filehash files cannot be read on the x64 system.

Byte order is not compatible at /usr/lib64/perl5/vendor_perl/5.16.3/x86_64-linux-thread-multi/ line 380, 
Byte order is not compatible at /usr/lib64/perl5/vendor_perl/5.16.3/x86_64-linux-thread-multi/ line 380, 
  at /var/www/foswiki/lib/Foswiki/Plugins/ line 1065

The solution is to use the nstore() and nfreeze() functions which use network order and are portable.

For an existing DGP install, either remove all of the working/work_areas/DirectedGraphPlugin/*-filehash files, or on the source system, use a small perl utility to read in each file using thaw() or retrieve() and then write them back out using nstore() or nfreeze().

-- GeorgeClark - 15 Dec 2013

There are a few more plugins that make use of Storable, i.e. DBCacheContrib. For maximum performance it generates caches in native store/freeze format. I have no real numbers comparing native vs cross-platform store formats, but compatibility for sure comes at a certain price. The trade off is between cleaning up working/* once as part of the migration process or having to pay the extra performance penalty (how large it ever might be) on a daily use. I actually prefer the former for DBCacheContrib as Storable is being used quite heavily. For DirectedGraphPlugin it might be okay as its use of Storable isn't as critical, i guess.

-- MichaelDaum - 16 Dec 2013

I'll check in a small perl script that will rewrite the -filehash files using nstore. This could be used in a migration. Run utility on the ia32 system, then copy to the x64 system. The issue with the DirectedGraphs is that some of them can run an extremely long time to generate, so loosing the cached versions could be quite expensive.

Well, when in doubt, bring data. ... It's counter intuitive, but nfreeze is a bit faster than freeze, and generates a smaller results.

Length: FROZEN 15958 NFROZEN 15949

     Rate   a   b
a 23635/s  -- -3%
b 24355/s  3%  --

Code to test was:
my $frozen = Storable::freeze( \%hash );
my $nfrozen = Storable::nfreeze( \%hash );
print STDERR "Length FROZEN " . length($frozen) . " NFROZEN " . length($nfrozen) . "\n";
    print "COMPARING\n";
    use Benchmark qw( cmpthese ) ;
    use Storable qw(freeze nfreeze);
    cmpthese( -20, {
        a => sub{
           my $X = Storable::freeze( \%hash );

        b => sub{
           my $X = Storable::nfreeze( \%hash );
    } );

-- GeorgeClark - 17 Dec 2013

Hm, seems you are not the first to find a performance & caching advantage in nfreeze:

Maybe I should experiment with it using big day-to-day data structures...

-- MichaelDaum - 18 Dec 2013
Topic revision: r10 - 21 Dec 2013, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy