You are here: Foswiki>Tasks Web>Item13432 (29 May 2015, CrawfordCurrie)Edit Attach

Item13432: RCSLite fails with Malformed UTF-8 character

Priority: Urgent
Current State: Closed
Released In: 1.2.0
Target Release: minor

Applies To: Engine
Component: FoswikiStore
Branches: master

Reported By: GeorgeClark
Waiting For:
Last Change By: CrawfordCurrie

Unable to "manage" an existing attachment with RcsLite. RcsWrap works fine.

Malformed UTF-8 character (unexpected non-continuation byte 0x40, immediately after start byte 0xc3) in string eq at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/RcsLiteHandler.pm line 171.
 at /usr/lib/perl5/vendor_perl/5.20.1/CGI/Carp.pm line 379.
   CGI::Carp::realdie("Malformed UTF-8 character (unexpected non-continuation byte 0"...) called at /usr/lib/perl5/vendor_perl/5.20.1/CGI/Carp.pm line 468
   CGI::Carp::die("Malformed UTF-8 character (unexpected non-continuation byte 0"...) called at /var/www/foswiki/distro/core/lib/AssertOn.pm line 15
   Assert::__ANON__("Malformed UTF-8 character (unexpected non-continuation byte 0"...) called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/RcsLiteHandler.pm line 171
   Foswiki::Store::Rcs::RcsLiteHandler::_readTo(GLOB(0x9154588), "\@") called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/RcsLiteHandler.pm line 287
   Foswiki::Store::Rcs::RcsLiteHandler::_ensureRead(Foswiki::Store::Rcs::RcsLiteHandler=HASH(0x9111af0), 3, 0) called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/RcsLiteHandler.pm line 633
   Foswiki::Store::Rcs::RcsLiteHandler::getInfo(Foswiki::Store::Rcs::RcsLiteHandler=HASH(0x9111af0), 3) called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/Store.pm line 310
   Foswiki::Store::Rcs::Store::_getAttachmentVersionInfo(Foswiki::Store::RcsLite=HASH(0x8ae7230), Foswiki::Meta=HASH(0x90996f8), 3, "CADEMIA_3.0-RC2.zip") called at /var/www/foswiki/distro/core/lib/Foswiki/Store/Rcs/Store.pm line 320
   Foswiki::Store::Rcs::Store::getVersionInfo(Foswiki::Store::RcsLite=HASH(0x8ae7230), Foswiki::Meta=HASH(0x90996f8), 3, "CADEMIA_3.0-RC2.zip") called at /var/www/foswiki/distro/core/lib/Foswiki/Meta.pm line 1562
   Foswiki::Meta::getRevisionInfo(Foswiki::Meta=HASH(0x90996f8), "CADEMIA_3.0-RC2.zip", 3) called at /var/www/foswiki/distro/core/lib/Foswiki/Attach.pm line 168
   Foswiki::Attach::formatVersions(Foswiki::Attach=HASH(0x90e7350), Foswiki::Meta=HASH(0x90996f8), "size", 15191747, "attr", "", "path", "CADEMIA_3.0-RC2.zip", "user", ...) called at /var/www/foswiki/distro/core/lib/Foswiki/UI/Attach.pm line 95
   Foswiki::UI::Attach::attach(Foswiki=HASH(0x88ad2b0)) called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 374
   Foswiki::UI::__ANON__() called at /usr/lib/perl5/vendor_perl/5.20.1/Error.pm line 419
   eval {...} called at /usr/lib/perl5/vendor_perl/5.20.1/Error.pm line 411
   Error::subs::try(CODE(0x804dea8), HASH(0x88bf8d0)) called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 500
   Foswiki::UI::_execute(Foswiki::Request=HASH(0x88a5fd0), CODE(0x887cf90), "attach", 1) called at /var/www/foswiki/distro/core/lib/Foswiki/UI.pm line 326
   Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x88a5fd0)) called at /var/www/foswiki/distro/core/lib/Foswiki/Engine/CGI.pm line 99
   Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0x84470d8)) called at attach line 29.

-- GeorgeClark - 24 May 2015

I can upload the zipfile of the attachment and associated rcs files. It's big. >60 meg. RcsLite on 1.1.9 can also manage the attachment, all be it very slow.

-- GeorgeClark - 24 May 2015

Found a smaller attachment with same issue. (It seems I can't "manage" most binary attachments).

Item13432.tgz: Topic/Attachment that cannot be processed with RcsLite.

 /var/www/foswiki/distro/core (master)$ tar -tzvf Item13432.tgz 
-rw-r--r-- gac/users       635 2015-05-24 19:23 data/Sandbox/TestUpload.txt
-rw-r--r-- gac/users      1747 2015-05-24 19:23 data/Sandbox/TestUpload.txt,v
drwxr-xr-x gac/users         0 2013-01-25 12:21 pub/Sandbox/TestUpload/
-r--r--r-- gac/users      4070 2013-01-25 12:21 pub/Sandbox/TestUpload/woo%hoo.wav,v
-r--r--r-- gac/users      4070 2013-01-25 12:20 pub/Sandbox/TestUpload/woohoo.wav,v
-rw-r--r-- gac/users      3814 2013-01-25 12:21 pub/Sandbox/TestUpload/woo%hoo.wav
-rw-r--r-- gac/users      3814 2013-01-25 12:20 pub/Sandbox/TestUpload/woohoo.wav

-- GeorgeClark - 24 May 2015

I have a possible fix. I think the issue is that RCS files should always be read as bytes, and never go through utf-8 encoding/decoding. Things appear to work fine with this applied. But I'm not going to check it in, as I don't understand the consequences.

{state} = 'nocommav';
         return;
     }
-    binmode( $fh, $Foswiki::UNICODE ? ':utf8' : undef );
+    binmode( $fh ); #, $Foswiki::UNICODE ? ':utf8' : undef );
 
     my $state   = 'admin.head';    # reset to start
     my $term    = ';';

-- GeorgeClark - 27 May 2015

Y, that is not a fix. Working on it.

-- CrawfordCurrie - 27 May 2015

Can of worms opened, cleaned out and closed again.

-- Main.CrawfordCurrie - 29 May 2015 - 06:51

ItemTemplate edit

Summary	RCSLite fails with Malformed UTF-8 character
ReportedBy	GeorgeClark
Codebase	trunk
SVN Range
AppliesTo	Engine
Component	FoswikiStore
Priority	Urgent
CurrentState	Closed
WaitingFor
Checkins	distro:d4561adcd955 distro:5a8526729d60 distro:fac6135767e0
TargetRelease	minor
ReleasedIn	1.2.0
CheckinsOnBranches	master
trunkCheckins
masterCheckins	distro:d4561adcd955 distro:5a8526729d60 distro:fac6135767e0
ItemBranchCheckins
Release01x01Checkins