Item5990: WYSIWYG gives fatal error when saving UTF-8/Old editor works
Priority: Urgent
Current State: Closed
Released In: 1.1.0
Target Release: minor
When attempting to save Slovenian UTF-8 text, WYSIWYG editor dies with
TWiki detected an internal error - please check your TWiki logs and webserver logs for more information.
Cannot decode string with wide characters
Selecting the Pick, and then save (i.e. saving in the raw editor) works.
[Tue Sep 09 08:15:28 2008] [error] [client 10.193.21.87] TWiki is likely to have problems, referer: http://hsl-wiki/twiki
/bin/edit/PSTestandLP/TeamManagementMiraRa\xc5\xbeman?t=1220940797
UTF-8 not yet supported as site charset -TWiki is likely to have problems[Tue Sep 9 08:21:01 2008] -e: Wide character in
print at /var/www/twiki/lib/TWiki/UI.pm line 185.
Cannot decode string with wide characters at /usr/lib/perl/5.8/Encode.pm line 166.
at /usr/lib/perl/5.8/Encode.pm line 166
Encode::decode('utf8', '<p>\x{010c}lanice tima: Tatjana Dariš, Tina Ravnikar, Mira Ra...') called at /usr/lib/per
l/5.8/Encode.pm line 206
Encode::decode_utf8('<p>\x{010c}lanice tima: Tatjana Dariš, Tina Ravnikar, Mira Ra...') called at /var/www/twiki/
lib/TWiki/Plugins/WysiwygPlugin/HTML2TML.pm line 113
TWiki::Plugins::WysiwygPlugin::HTML2TML::convert('TWiki::Plugins::WysiwygPlugin::HTML2TML=HASH(0x8d2f214)', '<p>\x{010c}
lanice tima: Tatjana Dariš, Tina Ravnikar, Mira Ra...', 'HASH(0x91b4e94)') called at /var/www/twiki/lib/TWiki/Plug
ins/WysiwygPlugin.pm line 228
TWiki::Plugins::WysiwygPlugin::TranslateHTML2TML('<p>\x{010c}lanice tima: Tatjana Dariš, Tina Ravnikar, Mira Ra..
.', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP') called at /var/www/twiki/lib/TWiki/Plugins/WysiwygPlugin.pm lin
e 196
TWiki::Plugins::WysiwygPlugin::afterEditHandler('<!--WYSIWYG content - do not remove this comment, and never u...
', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP', 'TWiki::Meta=HASH(0x91018b8)') called at /var/www/twiki/lib/TWik
i/Plugin.pm line 266
TWiki::Plugin::invoke('TWiki::Plugin=HASH(0x8fe9504)', 'afterEditHandler', '<!--WYSIWYG content - do not remove t
his comment, and never u...', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP', 'TWiki::Meta=HASH(0x91018b8)') called
at /var/www/twiki/lib/TWiki/Plugins.pm line 344
TWiki::Plugins::_dispatch('TWiki::Plugins=HASH(0x8d1b7a8)', 'afterEditHandler', '<!--WYSIWYG content - do not rem
ove this comment, and never u...', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP', 'TWiki::Meta=HASH(0x91018b8)') c
alled at /var/www/twiki/lib/TWiki/Plugins.pm line 647
TWiki::Plugins::afterEditHandler('TWiki::Plugins=HASH(0x8d1b7a8)', '<!--WYSIWYG content - do not remove this comm
ent, and never u...', 'TeamManagementMiraRa\x{c5}\x{be}man', 'PSTestandLP', 'TWiki::Meta=HASH(0x91018b8)') called at /var
/www/twiki/lib/TWiki/UI/Save.pm line 542
TWiki::UI::Save::save('TWiki=HASH(0x8d16090)') called at /var/www/twiki/lib/TWiki/UI.pm line 159
TWiki::UI::__ANON__() called at /usr/share/perl5/Error.pm line 380
eval {...} called at /usr/share/perl5/Error.pm line 372
Error::subs::try('CODE(0x8b064b8)', 'HASH(0x8d1b958)') called at /var/www/twiki/lib/TWiki/UI.pm line 197
TWiki::UI::run('CODE(0x90bac00)', 'save', 1) called at /var/www/twiki/bin/save line 32
ModPerl::ROOT::ModPerl::Registry::var_www_twiki_bin_save::handler('Apache2::RequestRec=SCALAR(0x9165068)') called
at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 203
eval {...} called at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 203
ModPerl::RegistryCooker::run('ModPerl::Registry=HASH(0x912e124)') called at /usr/lib/perl5/ModPerl/RegistryCooker
.pm line 169
ModPerl::RegistryCooker::default_handler('ModPerl::Registry=HASH(0x912e124)') called at /usr/lib/perl5/ModPerl/Re
gistry.pm line 30
ModPerl::Registry::handler('ModPerl::Registry', 'Apache2::RequestRec=SCALAR(0x9165068)') called at -e line 0
eval {...} called
[Tue Sep 9 08:21:01 2008] -e: Wide character in print at /var/www/twiki/lib/TWiki.pm line 1730.
Note: The same error came up as I was trying to save this bug report, so whatever version
is running the BUG reporting web, has this error too...
--
TWiki:Main/MatijaGrabnar - 10 Sep 2008
This also fails:
- raw edit
- paste this text:
<sticky>♀</sticky>
- save
- edit
- switchToRaw
Error:
Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.
Wide character in subroutine entry
Trunk and release branch
Using out-of-the-box encoding/locale settings
Evidently
Encode::decode()
is croaking when it encounters UTF-8 in
returnRESTResult
in
WysiwygPlugin.pm
, line 773 (release branch).
I added Crawford and Michael, who are free to remove themselves, I am wondering if this "urgent" because it results in data loss (potentially, although you can save from raw mode).
I will be investigating whether we can stop
HTML2TML from touching stuff inside
<sticky>
once and for all.
--
PaulHarvey - 14 Jan 2010
I would say that getting the code fixed so you at least do not loose data you have just types and want to save it what is important to fix.
Fixing so the UTF characters correctly save is less important with respect to urgentcy.
We need something in the code so it at least survives.
I can see we have had this bug since TWiki days so I will not let is block 1.0.9 but should be seen as urgent for 1.1.
--
KennethLavrsen - 14 Jan 2010
Commenting out:
# Convert (safe) named entities back to the
# site charset. Numeric entities are mapped straight to the
# corresponding code point unless their value overflow.
#require HTML::Entities;
#HTML::Entities::_decode_entities( $text, WC::safeEntities() );
"cures" my problem, in that I no longer need the sticky tag, but only if I don't switchToRaw (just save from WYSIWYG).
Using sticky tag does strange things (but at least does not crash)
--
PaulHarvey - 14 Jan 2010
Perhaps a try/catch around the offending call to
HTML::Entities
would be a simple enough way to avoid losing data...
--
PaulHarvey - 14 Jan 2010
This doesn't fail for me (trunk) though it does screw around with the entities
--
CrawfordCurrie - 17 Jan 2010
Not sure if this is the same problem, but on trunk, I've encountered a hard failure resulting is an empty topic after save. Rev 6097
Topic contains a set to use greek letters in the explicit numbering plugin:
* #Set EXPLICITNUMBERINGPLUGIN_ALPHASEQ = α,β,γ,δ,ε,ζ,η,θ,ι,κ,λ,μ
Editing the topic shows the entities in the editor as written above. They are not converted to wide characters. Save results in:
During save of ExplicitNumberingPlugin an error was found by the version control system. Please notify your Foswiki administrator.
Wide character in print at /var/www/SVN/foswiki/core/lib/Foswiki/Store/VC/Handler.pm line 811.
at /usr/lib/perl5/vendor_perl/5.10.1/CGI/Carp.pm line 354
CGI::Carp::realdie('Wide character in print at /var/www/SVN/foswiki/core/lib/Fosw...') called at /usr/lib/perl5/vendor_perl/5.10.1/CGI/Carp.pm line 435
CGI::Carp::die('Wide character in print at /var/www/SVN/foswiki/core/lib/Fosw...') called at /var/www/SVN/foswiki/core/lib/Foswiki.pm line 145
Foswiki::__ANON__('Wide character in print at /var/www/SVN/foswiki/core/lib/Fosw...') called at /var/www/SVN/foswiki/core/lib/Foswiki/Store/VC/Handler.pm line 811
Foswiki::Store::VC::Handler::saveFile('Foswiki::Store::VC::RcsWrapHandler=HASH(0x8b43ec0)', '/var/www/SVN/foswiki/core/data/System/ExplicitNumberingPlugin...', '%META:TOPICINFO{author="BaseUserMapping_333" date="1263952009...') called at /var/www/SVN/foswiki/core/lib/Foswiki/Store/VC/RcsWrapHandler.pm line 115
Foswiki::Store::VC::RcsWrapHandler::addRevisionFromText('Foswiki::Store::VC::RcsWrapHandler=HASH(0x8b43ec0)', '%META:TOPICINFO{author="BaseUserMapping_333" date="1263952009...', 'save topic', 'BaseUserMapping_333', undef) called at /var/www/SVN/foswiki/core/lib/Foswiki/Store/VC/Store.pm line 253
Foswiki::Store::VC::Store::saveTopic('Foswiki::Store::RcsWrap=HASH(0x84b0fc0)', 'Foswiki::Meta=HASH(0x8b4bca8)', 'BaseUserMapping_333', 'HASH(0x8240138)') called at /var/www/SVN/foswiki/core/lib/Foswiki/Meta.pm line 1599
Foswiki::Meta::__ANON__() called at /var/www/SVN/foswiki/core/lib/CPAN/lib//Error.pm line 379
Clicking the WikiText button from edit results in:
Wide character in subroutine entry at /usr/lib/perl5/5.10.1/i686-linux/Encode.pm line 174.
at /usr/lib/perl5/5.10.1/i686-linux/Encode.pm line 174
Encode::decode('iso-8859-1', '---+ <nop>%TOPIC%
Use the ==#<nop>#.==, ==#<nop>#..== _etc._...', 264) called at /var/www/SVN/foswiki/core/lib/Foswiki/Plugins/WysiwygPlugin/Handlers.pm line 600
Foswiki::Plugins::WysiwygPlugin::Handlers::returnRESTResult('Foswiki::Response=HASH(0x84afb08)', 200, '---+ <nop>%TOPIC%
Use the ==#<nop>#.==, ==#<nop>#..== _etc._...') called at /var/www/SVN/foswiki/core/lib/Foswiki/Plugins/WysiwygPlugin/Handlers.pm line 701
Foswiki::Plugins::WysiwygPlugin::Handlers::_restHTML2TML('Foswiki=HASH(0x84afa58)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x84afb08)') called at /var/www/SVN/foswiki/core/lib/Foswiki/Plugins/WysiwygPlugin.pm line 85
Foswiki::Plugins::WysiwygPlugin::_execute('_restHTML2TML', 'Foswiki=HASH(0x84afa58)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x84afb08)') called at /var/www/SVN/foswiki/core/lib/Foswiki/Plugins/WysiwygPlugin.pm line 69
Foswiki::Plugins::WysiwygPlugin::__ANON__('Foswiki=HASH(0x84afa58)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x84afb08)') called at /var/www/SVN/foswiki/core/lib/Foswiki/Func.pm line 2297
Foswiki::Func::__ANON__('Foswiki=HASH(0x84afa58)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x84afb08)') called at /var/www/SVN/foswiki/core/lib/Foswiki/UI/Rest.pm line 241
Foswiki::UI::Rest::rest('Foswiki=HASH(0x84afa58)') called at /var/www/SVN/foswiki/core/lib/Foswiki/UI.pm line 303
On Release branch, rev 6097. Wysiwyg save works, but entities are changed to hex characters. The Wikitext button results in:
Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.
Wide character in subroutine entry
| 2010-01-20T01:54:48Z warning | Wide character in subroutine entry at /usr/lib/perl5/5.10.1/i686-linux/Encode.pm line 174.
at /usr/lib/perl5/5.10.1/i686-linux/Encode.pm line 174
Encode::decode('iso-8859-1', '---+ <nop>%TOPIC%
Use the ==#<nop>#.==, ==#<nop>#..== _etc._...', 264) called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/Plugins/WysiwygPlugin.pm line 773
Foswiki::Plugins::WysiwygPlugin::returnRESTResult('Foswiki::Response=HASH(0x833b3e0)', 200, '---+ <nop>%TOPIC%
Use the ==#<nop>#.==, ==#<nop>#..== _etc._...') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/Plugins/WysiwygPlugin.pm line 874
Foswiki::Plugins::WysiwygPlugin::_restHTML2TML('Foswiki=HASH(0x805e5e8)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x833b3e0)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/Func.pm line 2211
Foswiki::Func::__ANON__('Foswiki=HASH(0x805e5e8)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x833b3e0)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/UI/Rest.pm line 187
Foswiki::UI::Rest::rest('Foswiki=HASH(0x805e5e8)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/UI.pm line 304
Foswiki::UI::__ANON__() called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/CPAN/lib//Error.pm line 379
eval {...} called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/CPAN/lib//Error.pm line 371
Error::subs::try('CODE(0x81ab340)', 'HASH(0x8896aa8)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/UI.pm line 391
Foswiki::UI::_execute('Foswiki::Request=HASH(0x83627a8)', 'CODE(0x8362638)', 'rest', 1) called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/UI.pm line 275
Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x83627a8)') called at /var/www/SVN/foswiki/branches/Release01x00/core/lib/Foswiki/Engine/CGI.pm line 29
Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x8271f90)') called
|
-- GeorgeClark - 20 Jan 2010
Related:
-- PaulHarvey - 24 Jan 2010
I am somewhat in awe of what Foswiki already gets right with converting between TML and HTML, but clearly there is room for improvement. I am trying to capture what Foswiki should do at UnicodeCharactersInNonUtf8Encodings.
-- MichaelTempest - 15 Jun 2010
Fixed on trunk.
-- MichaelTempest - 23 Jun 2010