You are here: Foswiki>Tasks Web>Item13424 (12 Jun 2015, GeorgeClark)Edit Attach

Item13424: Malformed UTF-8 character (unexpected end of string) fails for some languages on some perl versions.

pencil
Priority: Normal
Current State: Confirmed
Released In: n/a
Target Release: n/a
Applies To: Engine
Component: PlatformPerl516
Branches: master
Reported By: GeorgeClark
Waiting For:
Last Change By: GeorgeClark
The failure appears to be related to perl versions. It's failing on trunk.foswiki.org, which is running perl 5.16.3. It fails for only a few languages:
  • Spanish
  • French
And it currently fails on 5.16.3 (tested on foswiki.org and local perlbrew), Works with 5.18.2, 5.20.1 and 5.22-RC1

Malformed UTF-8 character (unexpected end of string) in string ne at /usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538.
 at /usr/local/lib/perl5/5.16/CGI/Carp.pm line 379.
   CGI::Carp::realdie("Malformed UTF-8 character (unexpected end of string) in strin"...) called at /usr/local/lib/perl5/5.16/CGI/Carp.pm line 468
   CGI::Carp::die("Malformed UTF-8 character (unexpected end of string) in strin"...) called at /usr/home/trunk.foswiki.org/core/lib/AssertOn.pm line 15
   Assert::__ANON__("Malformed UTF-8 character (unexpected end of string) in strin"...) called at /usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538
   Locale::Maketext::_compile(Foswiki::I18N::fr=HASH(0x804c38ac8), "Des id\x{e9}es, requ\x{ea}tes ou probl\x{e8}mes concernant [_1]\x{a0}? <a href='["...) called at /usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 224
   Locale::Maketext::maketext("<nop>Foswiki.org development testing", "mailto:webmaster\@foswiki.org?subject=Foswiki.org development "...) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/I18N.pm line 272
   Foswiki::I18N::maketext(Foswiki::I18N::fr=HASH(0x804c38ac8), "Ideas, requests, problems regarding [_1]? <a href='[_2]'>Send"..., "<nop>Foswiki.org development testing", "mailto:webmaster\@foswiki.org?subject=Foswiki.org development "...) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Macros/MAKETEXT.pm line 57
   Foswiki::MAKETEXT(Foswiki=HASH(0x802e41ea0), Foswiki::Attrs=HASH(0x80536f258), Foswiki::Meta=HASH(0x80565a5a0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3405
   Foswiki::_expandMacroOnTopicRendering(Foswiki=HASH(0x802e41ea0), "MAKETEXT", "\"Ideas, requests, problems regarding [_1]? <a href='[_2]'>Sen"..., Foswiki::Meta=HASH(0x80565a5a0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3267
   Foswiki::_processMacros(Foswiki=HASH(0x802e41ea0), "<span class=\"foswikiRight\"> <a href=\"http://foswiki.org/\"><im"..., CODE(0x802d845a0), Foswiki::Meta=HASH(0x80565a5a0), 15) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3285
   Foswiki::_processMacros(Foswiki=HASH(0x802e41ea0), "%WEBCOPYRIGHT%", CODE(0x802d845a0), Foswiki::Meta=HASH(0x80565a5a0), 16) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3064
   Foswiki::innerExpandMacros(Foswiki=HASH(0x802e41ea0), SCALAR(0x8051e6030), Foswiki::Meta=HASH(0x80565a5a0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Macros/INCLUDE.pm line 300
   Foswiki::__ANON__() called at /usr/local/lib/perl5/site_perl/5.16/Error.pm line 421
   eval {...} called at /usr/local/lib/perl5/site_perl/5.16/Error.pm line 413
   Error::subs::try(CODE(0x805659888), HASH(0x8057b5690)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Macros/INCLUDE.pm line 356
   Foswiki::_includeTopic(Foswiki=HASH(0x802e41ea0), Foswiki::Meta=HASH(0x80370e7c8), HASH(0x8051e7558), Foswiki::Attrs=HASH(0x80567d528)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Macros/INCLUDE.pm line 415
   Foswiki::INCLUDE(Foswiki=HASH(0x802e41ea0), Foswiki::Attrs=HASH(0x80567d528), Foswiki::Meta=HASH(0x80370e7c8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3405
   Foswiki::_expandMacroOnTopicRendering(Foswiki=HASH(0x802e41ea0), "INCLUDE", "\"System.WebBottomBar, System.WebBottomBarExample\"", Foswiki::Meta=HASH(0x80370e7c8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3267
   Foswiki::_processMacros(Foswiki=HASH(0x802e41ea0), " %IF{\"\$raw='on'\" then='<div class=\"patternSigLine clear\"><spa"..., CODE(0x802d845a0), Foswiki::Meta=HASH(0x80370e7c8), 16) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3064
   Foswiki::innerExpandMacros(Foswiki=HASH(0x802e41ea0), SCALAR(0x802d956c0), Foswiki::Meta=HASH(0x80370e7c8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki.pm line 3590
   Foswiki::expandMacros(Foswiki=HASH(0x802e41ea0), " %IF{\"\$raw='on'\" then='<div class=\"patternSigLine clear\"><spa"..., Foswiki::Meta=HASH(0x80370e7c8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Meta.pm line 3353
   Foswiki::Meta::expandMacros(Foswiki::Meta=HASH(0x80370e7c8), " %IF{\"\$raw='on'\" then='<div class=\"patternSigLine clear\"><spa"...) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI/View.pm line 479
   Foswiki::UI::View::_prepare(" %IF{\"\$raw='on'\" then='<div class=\"patternSigLine clear\"><spa"..., Foswiki::Meta=HASH(0x80370e7c8), 0) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI/View.pm line 465
   Foswiki::UI::View::view(Foswiki=HASH(0x802e41ea0)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI.pm line 374
   Foswiki::UI::__ANON__() called at /usr/local/lib/perl5/site_perl/5.16/Error.pm line 421
   eval {...} called at /usr/local/lib/perl5/site_perl/5.16/Error.pm line 413
   Error::subs::try(CODE(0x8010914f8), HASH(0x802e41978)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI.pm line 500
   Foswiki::UI::_execute(Foswiki::Request=HASH(0x802e196a8), CODE(0x802c3be28), "view", 1) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/UI.pm line 326
   Foswiki::UI::handleRequest(Foswiki::Request=HASH(0x802e196a8)) called at /usr/home/trunk.foswiki.org/core/lib/Foswiki/Engine/CGI.pm line 99
   Foswiki::Engine::CGI::run(Foswiki::Engine::CGI=HASH(0x80218d4b0)) called at /home/trunk.foswiki.org/core/bin/view line 29.

-- GeorgeClark - 22 May 2015

The following seems fixes it, but it is far from the optimal solution.

Principe, do the decode not at the Locale::Maketext::Lexicon level but higher, in the Foswiki's %MAKETEXT% macro level.

Currently, the Lexicon returns unicode characters, e.g. doing the decode_utf8 internally (and seems, something is wrong).

After the following patch, the Lexicon returns utf8 encoded byte string and Foswiki will do the decode. So moving the borders a bit closer into Foswiki. Not optimal, because for the same string will do more times the decode_utf8 but solves the problem. (at least in my test - needs more testing ).

--- refmast/foswiki/core/lib/Foswiki/I18N.pm   2015-05-21 14:20:40.000000000 +0200
+++ devmast/foswiki/core/lib/Foswiki/I18N.pm   2015-05-22 11:19:45.000000000 +0200
@@ -80,7 +80,7 @@
         unless (
             eval {
                 Locale::Maketext::Lexicon->import(
-                    { _decode => 1, $lang => [ Gettext => $langFile ] } );
+                    { $lang => [ Gettext => $langFile ] } );
                 1;
             }
           )
--- refmast/foswiki/core/lib/Foswiki/Macros/MAKETEXT.pm   2015-04-11 16:42:03.000000000 +0200
+++ devmast/foswiki/core/lib/Foswiki/Macros/MAKETEXT.pm   2015-05-22 11:36:56.000000000 +0200
@@ -54,7 +54,7 @@
     }
 
     # do the magic:
-    my $result = $this->i18n->maketext( $str, @args );
+    my $result = Encode::decode_utf8($this->i18n->maketext( $str, @args ));
 
     # replace accesskeys:
     $result =~

Also, the above would needed to do at every place, where the i18n->maketext is called.

-- JozefMojzis - 22 May 2015

I think we can downgrade this. I have not seen any occurrences of this since we changed the site to utf-8. Since thats the default now, and Actually they are still happening. But this seems to be a perl issue, marking this Normal, so its not a release blocker.

-- GeorgeClark - 12 Jun 2015
 

ItemTemplate edit

Summary Malformed UTF-8 character (unexpected end of string) fails for some languages on some perl versions.
ReportedBy GeorgeClark
Codebase trunk
SVN Range
AppliesTo Engine
Component PlatformPerl516
Priority Normal
CurrentState Confirmed
WaitingFor
Checkins distro:5cba582483a5
TargetRelease n/a
ReleasedIn n/a
CheckinsOnBranches master
trunkCheckins
masterCheckins distro:5cba582483a5
ItemBranchCheckins
Release01x01Checkins
Topic revision: r4 - 12 Jun 2015, GeorgeClark - This page was cached on 29 Jul 2016 - 06:08.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License