You are here: Foswiki>Tasks Web>Item5990 (04 Oct 2010, KennethLavrsen)Edit Attach

Item5990: WYSIWYG gives fatal error when saving UTF-8/Old editor works

Priority: Urgent
Current State: Closed
Released In: 1.1.0
Target Release: minor

Applies To: Extension
Component: WysiwygPlugin
Branches:

Reported By: TWiki:Main.MatijaGrabnar
Waiting For:
Last Change By: KennethLavrsen

When attempting to save Slovenian UTF-8 text, WYSIWYG editor dies with

TWiki detected an internal error - please check your TWiki logs and webserver logs for more information.

Cannot decode string with wide characters

Selecting the Pick, and then save (i.e. saving in the raw editor) works.

 [Tue Sep 09 08:15:28 2008] [error] [client 10.193.21.87] TWiki is likely to have problems, referer: http://hsl-wiki/twiki
/bin/edit/PSTestandLP/TeamManagementMiraRa\xc5\xbeman?t=1220940797
UTF-8 not yet supported as site charset -TWiki is likely to have problems[Tue Sep  9 08:21:01 2008] -e: Wide character in
 print at /var/www/twiki/lib/TWiki/UI.pm line 185.
Cannot decode string with wide characters at /usr/lib/perl/5.8/Encode.pm line 166.
 at /usr/lib/perl/5.8/Encode.pm line 166
        Encode::decode('utf8', '<p>\x{010c}lanice tima: Tatjana Dari&scaron;, Tina Ravnikar, Mira Ra...') called at /usr/lib/per
l/5.8/Encode.pm line 206
        Encode::decode_utf8('<p>\x{010c}lanice tima: Tatjana Dari&scaron;, Tina Ravnikar, Mira Ra...') called at /var/www/twiki/
lib/TWiki/Plugins/WysiwygPlugin/HTML2TML.pm line 113
        TWiki::Plugins::WysiwygPlugin::HTML2TML::convert('TWiki::Plugins::WysiwygPlugin::HTML2TML=HASH(0x8d2f214)', '<p>\x{010c}
lanice tima: Tatjana Dari&scaron;, Tina Ravnikar, Mira Ra...', 'HASH(0x91b4e94)') called at /var/www/twiki/lib/TWiki/Plug
ins/WysiwygPlugin.pm line 228
        TWiki::Plugins::WysiwygPlugin::TranslateHTML2TML('<p>\x{010c}lanice tima: Tatjana Dari&scaron;, Tina Ravnikar, Mira Ra..
.', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP') called at /var/www/twiki/lib/TWiki/Plugins/WysiwygPlugin.pm lin
e 196
        TWiki::Plugins::WysiwygPlugin::afterEditHandler('<!--WYSIWYG content - do not remove this comment, and never u...
', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP', 'TWiki::Meta=HASH(0x91018b8)') called at /var/www/twiki/lib/TWik
i/Plugin.pm line 266
        TWiki::Plugin::invoke('TWiki::Plugin=HASH(0x8fe9504)', 'afterEditHandler', '<!--WYSIWYG content - do not remove t
his comment, and never u...', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP', 'TWiki::Meta=HASH(0x91018b8)') called
 at /var/www/twiki/lib/TWiki/Plugins.pm line 344
        TWiki::Plugins::_dispatch('TWiki::Plugins=HASH(0x8d1b7a8)', 'afterEditHandler', '<!--WYSIWYG content - do not rem
ove this comment, and never u...', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP', 'TWiki::Meta=HASH(0x91018b8)') c
alled at /var/www/twiki/lib/TWiki/Plugins.pm line 647
        TWiki::Plugins::afterEditHandler('TWiki::Plugins=HASH(0x8d1b7a8)', '<!--WYSIWYG content - do not remove this comm
ent, and never u...', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP', 'TWiki::Meta=HASH(0x91018b8)') called at /var
/www/twiki/lib/TWiki/UI/Save.pm line 542
        TWiki::UI::Save::save('TWiki=HASH(0x8d16090)') called at /var/www/twiki/lib/TWiki/UI.pm line 159
        TWiki::UI::__ANON__() called at /usr/share/perl5/Error.pm line 380
        eval {...} called at /usr/share/perl5/Error.pm line 372
        Error::subs::try('CODE(0x8b064b8)', 'HASH(0x8d1b958)') called at /var/www/twiki/lib/TWiki/UI.pm line 197
        TWiki::UI::run('CODE(0x90bac00)', 'save', 1) called at /var/www/twiki/bin/save line 32
        ModPerl::ROOT::ModPerl::Registry::var_www_twiki_bin_save::handler('Apache2::RequestRec=SCALAR(0x9165068)') called
 at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 203
        eval {...} called at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 203
        ModPerl::RegistryCooker::run('ModPerl::Registry=HASH(0x912e124)') called at /usr/lib/perl5/ModPerl/RegistryCooker
.pm line 169
        ModPerl::RegistryCooker::default_handler('ModPerl::Registry=HASH(0x912e124)') called at /usr/lib/perl5/ModPerl/Re
gistry.pm line 30
        ModPerl::Registry::handler('ModPerl::Registry', 'Apache2::RequestRec=SCALAR(0x9165068)') called at -e line 0
        eval {...} called
[Tue Sep  9 08:21:01 2008] -e: Wide character in print at /var/www/twiki/lib/TWiki.pm line 1730.

Note: The same error came up as I was trying to save this bug report, so whatever version is running the BUG reporting web, has this error too...

-- TWiki:Main/MatijaGrabnar - 10 Sep 2008

ColasNahaboo - 24 Nov 2008: On http://develop.twiki.org/~twiki4/cgi-bin/view/Bugs/Item5990: anoter user had the same problem, but CDot couldn't reproduce

This also fails:

raw edit
paste this text:
<sticky>♀</sticky>
save
edit
switchToRaw

Error:

Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.

Wide character in subroutine entry

Trunk and release branch

Using out-of-the-box encoding/locale settings

Item5990_apache_backtrace.txt: apache backtrace release r6046

Evidently Encode::decode() is croaking when it encounters UTF-8 in returnRESTResult in WysiwygPlugin.pm, line 773 (release branch).

I added Crawford and Michael, who are free to remove themselves, I am wondering if this "urgent" because it results in data loss (potentially, although you can save from raw mode).

I will be investigating whether we can stop HTML2TML from touching stuff inside <sticky> once and for all.

-- PaulHarvey - 14 Jan 2010

I would say that getting the code fixed so you at least do not loose data you have just types and want to save it what is important to fix.

Fixing so the UTF characters correctly save is less important with respect to urgentcy.

We need something in the code so it at least survives.

I can see we have had this bug since TWiki days so I will not let is block 1.0.9 but should be seen as urgent for 1.1.

-- KennethLavrsen - 14 Jan 2010

Commenting out:

    # Convert (safe) named entities back to the
    # site charset. Numeric entities are mapped straight to the
    # corresponding code point unless their value overflow.
    #require HTML::Entities;
    #HTML::Entities::_decode_entities( $text, WC::safeEntities() );

"cures" my problem, in that I no longer need the sticky tag, but only if I don't switchToRaw (just save from WYSIWYG).

Using sticky tag does strange things (but at least does not crash)

-- PaulHarvey - 14 Jan 2010

Perhaps a try/catch around the offending call to HTML::Entities would be a simple enough way to avoid losing data...

-- PaulHarvey - 14 Jan 2010

This doesn't fail for me (trunk) though it does screw around with the entities

-- CrawfordCurrie - 17 Jan 2010

Not sure if this is the same problem, but on trunk, I've encountered a hard failure resulting is an empty topic after save. Rev 6097

Topic contains a set to use greek letters in the explicit numbering plugin:

   * #Set EXPLICITNUMBERINGPLUGIN_ALPHASEQ = &#945;,&#946;,&#947;,&#948;,&#949;,&#950;,&#951;,&#952;,&#953;,&#954;,&#955;,&#956;

Editing the topic shows the entities in the editor as written above. They are not converted to wide characters. Save results in:

During save of ExplicitNumberingPlugin an error was found by the version control system. Please notify your Foswiki administrator.

Wide character in print at /var/www/SVN/foswiki/core/lib/Foswiki/Store/VC/Handler.pm line 811. at /usr/lib/perl5/vendor_perl/5.10.1/CGI/Carp.pm line 354 CGI::Carp::realdie('Wide character in print at /var/www/SVN/foswiki/core/lib/Fosw...') called at /usr/lib/perl5/vendor_perl/5.10.1/CGI/Carp.pm line 435 CGI::Carp::die('Wide character in print at /var/www/SVN/foswiki/core/lib/Fosw...') called at /var/www/SVN/foswiki/core/lib/Foswiki.pm line 145 Foswiki::__ANON__('Wide character in print at /var/www/SVN/foswiki/core/lib/Fosw...') called at /var/www/SVN/foswiki/core/lib/Foswiki/Store/VC/Handler.pm line 811 Foswiki::Store::VC::Handler::saveFile('Foswiki::Store::VC::RcsWrapHandler=HASH(0x8b43ec0)', '/var/www/SVN/foswiki/core/data/System/ExplicitNumberingPlugin...', '%META:TOPICINFO{author="BaseUserMapping_333" date="1263952009...') called at /var/www/SVN/foswiki/core/lib/Foswiki/Store/VC/RcsWrapHandler.pm line 115 Foswiki::Store::VC::RcsWrapHandler::addRevisionFromText('Foswiki::Store::VC::RcsWrapHandler=HASH(0x8b43ec0)', '%META:TOPICINFO{author="BaseUserMapping_333" date="1263952009...', 'save topic', 'BaseUserMapping_333', undef) called at /var/www/SVN/foswiki/core/lib/Foswiki/Store/VC/Store.pm line 253 Foswiki::Store::VC::Store::saveTopic('Foswiki::Store::RcsWrap=HASH(0x84b0fc0)', 'Foswiki::Meta=HASH(0x8b4bca8)', 'BaseUserMapping_333', 'HASH(0x8240138)') called at /var/www/SVN/foswiki/core/lib/Foswiki/Meta.pm line 1599 Foswiki::Meta::__ANON__() called at /var/www/SVN/foswiki/core/lib/CPAN/lib//Error.pm line 379

Clicking the WikiText button from edit results in:

Wide character in subroutine entry at /usr/lib/perl5/5.10.1/i686-linux/Encode.pm line 174.
 at /usr/lib/perl5/5.10.1/i686-linux/Encode.pm line 174
   Encode::decode('iso-8859-1', '---+ <nop>%TOPIC%

Use the ==#<nop>#.==, ==#<nop>#..== _etc._...', 264) called at /var/www/SVN/foswiki/core/lib/Foswiki/Plugins/WysiwygPlugin/Handlers.pm line 600
   Foswiki::Plugins::WysiwygPlugin::Handlers::returnRESTResult('Foswiki::Response=HASH(0x84afb08)', 200, '---+ <nop>%TOPIC%
Use the ==#<nop>#.==, ==#<nop>#..== _etc._...') called at /var/www/SVN/foswiki/core/lib/Foswiki/Plugins/WysiwygPlugin/Handlers.pm line 701
   Foswiki::Plugins::WysiwygPlugin::Handlers::_restHTML2TML('Foswiki=HASH(0x84afa58)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x84afb08)') called at /var/www/SVN/foswiki/core/lib/Foswiki/Plugins/WysiwygPlugin.pm line 85
   Foswiki::Plugins::WysiwygPlugin::_execute('_restHTML2TML', 'Foswiki=HASH(0x84afa58)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x84afb08)') called at /var/www/SVN/foswiki/core/lib/Foswiki/Plugins/WysiwygPlugin.pm line 69
   Foswiki::Plugins::WysiwygPlugin::__ANON__('Foswiki=HASH(0x84afa58)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x84afb08)') called at /var/www/SVN/foswiki/core/lib/Foswiki/Func.pm line 2297
   Foswiki::Func::__ANON__('Foswiki=HASH(0x84afa58)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x84afb08)') called at /var/www/SVN/foswiki/core/lib/Foswiki/UI/Rest.pm line 241
   Foswiki::UI::Rest::rest('Foswiki=HASH(0x84afa58)') called at /var/www/SVN/foswiki/core/lib/Foswiki/UI.pm line 303

On Release branch, rev 6097. Wysiwyg save works, but entities are changed to hex characters. The Wikitext button results in:

Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.

Wide character in subroutine entry


| 2010-01-20T01:54:48Z warning | Wide character in subroutine entry at /usr/lib/perl5/5.10.1/i686-linux/Encode.pm line 174.
 at /usr/lib/perl5/5.10.1/i686-linux/Encode.pm line 174
        Encode::decode('iso-8859-1', '---+ <nop>%TOPIC%

Use the ==#<nop>#.==, ==#<nop>#..== _etc._...', 264) called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/Plugins/WysiwygPlugin.pm line 773
        Foswiki::Plugins::WysiwygPlugin::returnRESTResult('Foswiki::Response=HASH(0x833b3e0)', 200, '---+ <nop>%TOPIC%

Use the ==#<nop>#.==, ==#<nop>#..== _etc._...') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/Plugins/WysiwygPlugin.pm line 874
        Foswiki::Plugins::WysiwygPlugin::_restHTML2TML('Foswiki=HASH(0x805e5e8)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x833b3e0)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/Func.pm line 2211
        Foswiki::Func::__ANON__('Foswiki=HASH(0x805e5e8)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x833b3e0)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/UI/Rest.pm line 187
        Foswiki::UI::Rest::rest('Foswiki=HASH(0x805e5e8)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/UI.pm line 304
        Foswiki::UI::__ANON__() called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/CPAN/lib//Error.pm line 379
        eval {...} called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/CPAN/lib//Error.pm line 371
        Error::subs::try('CODE(0x81ab340)', 'HASH(0x8896aa8)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/UI.pm line 391
        Foswiki::UI::_execute('Foswiki::Request=HASH(0x83627a8)', 'CODE(0x8362638)', 'rest', 1) called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/UI.pm line 275
        Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x83627a8)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/Engine/CGI.pm line 29
        Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x8271f90)') called
 |

-- GeorgeClark - 20 Jan 2010

-- PaulHarvey - 24 Jan 2010

I am somewhat in awe of what Foswiki already gets right with converting between TML and HTML, but clearly there is room for improvement. I am trying to capture what Foswiki should do at UnicodeCharactersInNonUtf8Encodings.

-- MichaelTempest - 15 Jun 2010

Fixed on trunk.

-- MichaelTempest - 23 Jun 2010

ItemTemplate edit

Summary	WYSIWYG gives fatal error when saving UTF-8/Old editor works
ReportedBy	TWiki:Main.MatijaGrabnar
Codebase	1.0.9, 1.0.8, trunk
SVN Range
AppliesTo	Extension
Component	WysiwygPlugin
Priority	Urgent
CurrentState	Closed
WaitingFor
Checkins	distro:cc4d235d29d5, distro:a820f29b54a4
TargetRelease	minor
ReleasedIn	1.1.0