Item13573: MultiTopicSavePlugin fails in 2.0 because of some incompatible changes in charsets ≤

I use the plugin MultiTopicSavePlugin extensively (Made it myself) and it works very well in 1.1.9

I am running on a Git master clone. I need to run the server so it is compatible with all my existing topics so I have chosen iso-8859-1 instead of UTF8. There is no way I can start converting 100000s of topics to UTF8 without risk of screwing up business critical information.

I also run with RCS Wrap.

The plugin fails when saving.

This is the error reported.

This is a real show stopper for me and I have no clue how to fix it in the plugin.

I am releasing a new version of MultiTopicSavePlugin now and it will not work with 2.0. Any fixes need to be compatible with 1.1.9 also

Even of this is a plugin related bug the problem is that 2.0 is not compatible with existing plugins. Something related to the UTF8 changes you did is broken.

Note that the plugin uses the rest interface. And I am using only the API in the plugin.

ERROR: (500) Internal server error - "\x{2264}" does not map to iso-8859-1 at /usr/lib64/perl5/vendor_perl/Encode.pm line 160.
 at /usr/lib64/perl5/vendor_perl/Encode.pm line 160.
   Encode::encode('iso-8859-1', '%META:TOPICINFO{author="c12179" comment="reprev" date="143810...', 1) called at /var/www/foswikidev/core/lib/Foswiki/Store.pm line 276
   Foswiki::Store::encode('%META:TOPICINFO{author="c12179" comment="reprev" date="143810...') called at /var/www/foswikidev/core/lib/Foswiki/Store/Rcs/Store.pm line 429
   Foswiki::Store::Rcs::Store::saveTopic('Foswiki::Store::RcsWrap=HASH(0x2224440)', 'Foswiki::Meta=HASH(0x32223f0)', 'c12179', 'HASH(0x1ca4df0)') called at /var/www/foswikidev/core/lib/Foswiki/Meta.pm line 2135
   Foswiki::Meta::__ANON__() called at /usr/share/perl5/vendor_perl/Error.pm line 419
   eval {...} called at /usr/share/perl5/vendor_perl/Error.pm line 411
   Error::subs::try('CODE(0x36cea20)', 'HASH(0x36cf020)') called at /var/www/foswikidev/core/lib/Foswiki/Meta.pm line 2152
   Foswiki::Meta::saveAs('Foswiki::Meta=HASH(0x32223f0)') called at /var/www/foswikidev/core/lib/Foswiki/Meta.pm line 1999
   Foswiki::Meta::__ANON__() called at /usr/share/perl5/vendor_perl/Error.pm line 419
   eval {...} called at /usr/share/perl5/vendor_perl/Error.pm line 411
   Error::subs::try('CODE(0x3475d48)', 'HASH(0x36c9d08)') called at /var/www/foswikidev/core/lib/Foswiki/Meta.pm line 2003
   Foswiki::Meta::save('Foswiki::Meta=HASH(0x32223f0)') called at /var/www/foswikidev/core/lib/Foswiki/Func.pm line 2115
   Foswiki::Func::saveTopic('Station/Track', 'HwSR00001151', 'Foswiki::Meta=HASH(0x3460ac8)', '---++ Status Updates\x{a}\x{a}_Assessor and Assignee_\x{a}\x{a}\x{a}---++++ Statu...') called at /var/www/foswikidev/core/lib/Foswiki/Plugins/MultiTopicSavePlugin.pm line 704
   Foswiki::Plugins::MultiTopicSavePlugin::restMultiTopicSave('Foswiki=HASH(0x2223b88)', 'MultiTopicSavePlugin', 'multitopicsave', 'Foswiki::Response=HASH(0x2223cc0)') called at /var/www/foswikidev/core/lib/Foswiki/Func.pm line 768
   Foswiki::Func::__ANON__('Foswiki=HASH(0x2223b88)', 'MultiTopicSavePlugin', 'multitopicsave', 'Foswiki::Response=HASH(0x2223cc0)') called at /var/www/foswikidev/core/lib/Foswiki/UI/Rest.pm line 275
   Foswiki::UI::Rest::__ANON__() called at /usr/share/perl5/vendor_perl/Error.pm line 419
   eval {...} called at /usr/share/perl5/vendor_perl/Error.pm line 411
   Error::subs::try('CODE(0x3425ea8)', 'HASH(0x32222b8)') called at /var/www/foswikidev/core/lib/Foswiki/UI/Rest.pm line 290
   Foswiki::UI::Rest::rest('Foswiki=HASH(0x2223b88)') called at /var/www/foswikidev/core/lib/Foswiki/UI.pm line 374
   Foswiki::UI::__ANON__() called at /usr/share/perl5/vendor_perl/Error.pm line 419
   eval {...} called at /usr/share/perl5/vendor_perl/Error.pm line 411
   Error::subs::try('CODE(0x13edf00)', 'HASH(0x23158c0)') called at /var/www/foswikidev/core/lib/Foswiki/UI.pm line 500
   Foswiki::UI::_execute('Foswiki::Request=HASH(0x13f57a8)', 'CODE(0x22ffb38)', 'rest', 1) called at /var/www/foswikidev/core/lib/Foswiki/UI.pm line 326
   Foswiki::UI::handleRequest('Foswiki::Request=HASH(0x13f57a8)') called at /var/www/foswikidev/core/lib/Foswiki/Engine/CGI.pm line 99
   Foswiki::Engine::CGI::run('Foswiki::Engine::CGI=HASH(0x1a858a8)') called at /var/www/foswikidev/core/bin/rest line 29.

-- KennethLavrsen - 28 Jul 2015

Update. The bug is triggered by having an html entity ≤ in a formfield. This content gets converted to UTF8. That does not happen in 1.1.9. It seems to be a rest interface related problem. I can edit and save the same topic just fine in the normal edit/save sequency.

-- KennethLavrsen - 28 Jul 2015

The content in a formfield is exactly this

Condition                                 ----                       RF_CPLD_RX_FE_SEL[1,2]
Rx freq ≤ FE_RANGE1         ----                                        11
FE_RANGE1 < Rx freq &#8804; FE_RANGE2     ----                     10
FE_RANGE2 < Rx freq &#8804; FE_RANGE3       ----                    01
FE_RANGE3 < RX freq                             ----                      00

So when sending this through the rest interface the ≤ gets translated to UTF somewhere and then the Encode for iso fails

-- KennethLavrsen - 28 Jul 2015

See also related Item13575

-- GeorgeClark - 28 Jul 2015

The update that Clarke did in Item13575 changed the behaviour of this bug. The plugin can now save topics with html entities that would have become utf8. BUT it is still not working. Everytime the plugin compares the old value with the new submitted it sees a difference so it saves the topic even though you did not touch it. That is a big problem.

The plugin reads a fields value with and uses this to display

    my ( $meta, undef ) = Foswiki::Func::readTopic( $web, $topic );
    my $value = $meta->get( 'FIELD', $field );
    my $returnvalue = defined $value ? $value->{'value'} : '';

When we compare the new value with the old, the old value is read with same meta->get

                my $fieldhashref = $meta->get( 'FIELD', $fieldName );
            
                $oldValue = $fieldhashref ?
                               $meta->get( 'FIELD', $fieldName )->{'value'} :
                               '';

Later the value is put back with

    $meta->putKeyed( 'FIELD', { name => $fieldName, value => $value } );

So the conversion must likely happens when the user submits the form via the rest interface.

the place the submit value is picked up is

    my @fieldValues = $query->multi_param($key);

For this plugin to ever work, The plugin needs to get the unchanged content in all 3 cases.

$meta->get, and $query->multi_param must return the content 100% identical of what I saved with $meta->putKeyed. If a conversion happens it must be 100% two way transparent.

-- KennethLavrsen - 29 Jul 2015

After a lot of analysis it is clear that the my @fieldValues = $query->multi_param($key); is returning encoded values. HTML entities get encoded to a value like \xe2\x89\xa4 for a ≤

The $meta->get( 'FIELD', $fieldName ); returns lean ≤

The $meta->putKeyed( 'FIELD', { name => $fieldName, value => $value } ); stores cleanly ≤

The Browser receives a clean ≤ from the server. I can see that by viewing sources.

Something is goofy with @fieldValues = $query->multi_param($key);

-- KennethLavrsen - 29 Jul 2015

Adding a little test case

---

This test case is simple. Click the edit. Note that there is a %le; in the Summary field of this Task. If you leave it untouched, the plugin should say that nothing changed and should not change this topic. When it fails it says that 1 topic is changed.

Edit Summary and lock the Item

MultiTopicSavePlugin fails in 2.0 because of some incompatible changes in charsets ≤

---

-- KennethLavrsen - 29 Jul 2015

Note that this test case does not fails here on foswiki.org for some reason. Perl version? UTF8 vs ISO?

-- KennethLavrsen - 29 Jul 2015

In 1.1.9 the ≤ gets converted to ≤ when using the plugin

This gets saved and from then on ≤ gets converted to itself and I do not see these ghost changes more than once. That is why it works. But there is still a conversion happening.

-- KennethLavrsen - 29 Jul 2015

Using the test case in this topic, along with firebug & wireshark, the culprit is the browser. I updated the summary to contain the ≤ entity. The MultiTopicSavePlugin created a textarea containing:

<textarea class='foswikiTextarea' cols='80' rows='1' name='multitopicsavefield{Tasks.Item13573}{Summary
}'>MultiTopicSavePlugin fails in 2.0 because of some incompatible changes in charsets  &le;</textarea
>

Upon clicking Save, the POST contained:

validation_key=78846875ccd9765b22f5fcf8a5a43922&redirectweb=Tasks&redirecttopic=Item13573&topic=Tasks
.Item13573&multitopicsavefield%7BTasks.Item13573%7D%7BSummary%7D=MultiTopicSavePlugin+fails+in+2.0+because
+of+some+incompatible+changes+in+charsets++%E2%89%A4

So the browser is converting the named entity into the hex entity.

-- GeorgeClark - 30 Jul 2015

Here is the "round trip"

1. the original form data contains literal ≤
2. ->> Sent to browser as </textarea> &le;  </textarea>
3. User presses submit, touching nothing
4. <<- POST contains %E2%89%A4
5. Foswiki automatically converts %E2%89%A4 into the unicode ≤
6. Plugin compares ≤ to &le; and fails

We can't reliably convert the posted ≤ into an entity, we don't really know if the original topic contained the named or numeric entity. But what you can do is convert the named entities into unicode characters. So continuing on...

1. Plugin does HTML::Entities::decode_entities(original formfield source)
2. Now plugin compares ≤ to ≤ and they should match.

And if they don't match, and plugin saves the topic:

• For a utf-8 store, the ≤ is written to the store as a unicode character.
• For an iso* store, after the patch to Item13575, the ≤ causes a utf8 exception, and HTML::Entities converts it to the named &le; entity.
• Prior to Item13757, Store would crash if the {Store}{Encoding} did not contain that character.

-- GeorgeClark - 30 Jul 2015

This task seems to be fixed. Just ran the above testcase again, and it did not save the topic, which is the behaviour you want, though I'm testing on a uft-8 store, Not sure what would happen on an iso-8859 store.

-- GeorgeClark - 24 Nov 2015

Can this be closed?

-- GeorgeClark - 24 Dec 2015

Commenting is disabled while not logged in