Item2311: WysiwygPlugin not compatible with word-break character

Priority:	CurrentState:	AppliesTo:	Component:	WaitingFor:
Normal	Closed	Extension	WysiwygPlugin

A​B

does not survive a roundtrip through WysiwygPlugin.

If I open a topic in TMCE, click WikiText, paste in the above text, click WYSIWYG, and click WikiText again, the text is converted to

A\x{200b}B

If I put that in a sticky block, like this:

<sticky>A&#8203;B</sticky>

into the WikiText editor, I get this when I click WYSIWYG and then click WikiText again:

Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information.

Wide character in subroutine entry

This was in the apache log:

[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] [Fri Oct 30 16:16:27 2009] rest: Wide character in print at /home/config/wiki/foswiki/lib/Foswiki/UI.pm line 347., referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] Wide character in subroutine entry at /usr/lib/perl/5.8/Encode.pm line 166., referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1]  at /usr/lib/perl/5.8/Encode.pm line 166, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tEncode::decode('iso-8859-1', 'A\xe2\x80\x8bB, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] ', 264) called at /home/config/wiki/foswiki/lib/Foswiki/Plugins/WysiwygPlugin.pm line 778, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tFoswiki::Plugins::WysiwygPlugin::returnRESTResult('Foswiki::Response=HASH(0x855efc0)', 200, 'A\xe2\x80\x8bB, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] ') called at /home/config/wiki/foswiki/lib/Foswiki/Plugins/WysiwygPlugin.pm line 879, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tFoswiki::Plugins::WysiwygPlugin::_restHTML2TML('Foswiki=HASH(0x8153648)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x855efc0)') called at /home/config/wiki/foswiki/lib/Foswiki/Func.pm line 2222, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tFoswiki::Func::__ANON__('Foswiki=HASH(0x8153648)', 'WysiwygPlugin', 'html2tml', 'Foswiki::Response=HASH(0x855efc0)') called at /home/config/wiki/foswiki/lib/Foswiki/UI/Rest.pm line 187, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tFoswiki::UI::Rest::rest('Foswiki=HASH(0x8153648)') called at /home/config/wiki/foswiki/lib/Foswiki/UI.pm line 299, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tFoswiki::UI::__ANON__() called at /usr/share/perl5/Error.pm line 380, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \teval {...} called at /usr/share/perl5/Error.pm line 372, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tError::subs::try('CODE(0x82f8d5c)', 'HASH(0x8c0aa64)') called at /home/config/wiki/foswiki/lib/Foswiki/UI.pm line 381, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tFoswiki::UI::_execute('Foswiki::Request=HASH(0x8531410)', 'CODE(0x8531368)', 'rest', 1) called at /home/config/wiki/foswiki/lib/Foswiki/UI.pm line 270, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tFoswiki::UI::handleRequest('Foswiki::Request=HASH(0x8531410)') called at /home/config/wiki/foswiki/lib/Foswiki/Engine/CGI.pm line 26, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] \tFoswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x83ec5bc)') called, referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821
[Fri Oct 30 16:16:27 2009] [error] [client 172.16.225.1] [Fri Oct 30 16:16:27 2009] rest: Wide character in print at /home/config/wiki/foswiki/lib/Foswiki/Logger/PlainFile.pm line 63., referer: http://172.16.225.134/foswiki/bin/edit/Sandbox/TestTopicAUTOINC0?t=160821

-- MichaelTempest - 30 Oct 2009

---

Bummer, Unicode issue. Trying to decode a string that already contains a wide-byte character. Add it to the pile of UTF8/Unicode issues.

-- CrawfordCurrie - 30 Oct 2009

Note that it breaks horribly on conversion from HTML back to TML. WysiwygPlugin correctly converts

<sticky>&#8203;</sticky>

to

<div class="WYSIWYG_STICKY">&amp;#8203;</div>

-- MichaelTempest - 30 Oct 2009

Fixed on trunk.

-- MichaelTempest - 20 Jun 2010