Item12569: Sorting in tableplugin is wrong in german

If i sort a table with TablePlugin in german umlaute are at the end of the table. This looks like:

a b c

x y z ä ö ü

but it should look like: a ä b c

ö

ü

x y z

-- FlorianPribahsnik - 23 Aug 2013

Same issue in our installation (Foswiki 1.1.8, UseLocale=1, Site::Locale=de_AT.ISO-8859-1): When using the column sort feature on a table like this http://foswiki.org/Sandbox/GermanTableSort the correct sort order would be A - Ä - B - ..., but instead the Umlauts show up at the very end of the table.

-- PhilippGortan - 24 Feb 2014

The patch seems to be rather trivial: When UseLocale is enabled in the foswiki config, then the specified locale should be used for sorting. The implementation is identical to Foswiki::Search, Foswiki::Render, etc.

Item12569.patch: Proposed patch

-- PhilippGortan - 24 Feb 2014

Unfortunately UseLocale creates all sorts of other problems together with perl's tainted check.

-- MichaelDaum - 25 Feb 2014

The attached script locale-test.pl tries to verify my assumption that:

• using "use locale" is scoped, so calling it in a block does not have any effect on the outer code
• "use locale" can thus be switched on and off at runtime without using the begin-if-require-import-hack
• outside of the block, \w does not taint the submatch

Can someone please verify that? Calling the script without params sorts without locale, adding any parameter when calling sorts with locale. In both cases, $res is not tainted.

-- PhilippGortan - 25 Feb 2014

Jan, Could you validate the addition of the UseLocale patch to TablePlugin/Core.pm? The proposed fix here is the same as what you used in just about every other module. Mark this waiting for release if it looks good. Thanks.

-- GeorgeClark - 12 Jan 2015

Definitely not fixed in 1.2.0. Note that client side sorting in javascript is also broken. In the core perl side, the fundamental issue is that we do not "normalize" the unicode characters. The following patch fixes the perl table sort implementation, but is NOT applied, as this needs much more discussion. Also the javascript side would need equivalent work.

diff --git a/TablePlugin/lib/Foswiki/Plugins/TablePlugin/Core.pm b/TablePlugin/lib/Foswiki/Plugins/TablePlugin/Core.pm
index 7764d25..9c9065a 100644
--- a/TablePlugin/lib/Foswiki/Plugins/TablePlugin/Core.pm
+++ b/TablePlugin/lib/Foswiki/Plugins/TablePlugin/Core.pm
@@ -10,6 +10,8 @@ use Foswiki::Plugins::TablePlugin ();
 use Foswiki::Time;
 use Error qw(:try);
 
+use Unicode::Normalize;
+
 BEGIN {
     if ( $Foswiki::cfg{UseLocale} ) {
         require locale;
@@ -1502,7 +1504,7 @@ sub emitTable {
             && $currentSortDirection == $SORT_DIRECTION->{'ASCENDING'} )
         {
             @curTable = sort {
-                     $a->[$sortCol]->{sortText} cmp $b->[$sortCol]->{sortText}
+                     NFKD($a->[$sortCol]->{sortText}) cmp NFKD($b->[$sortCol]->{sortText})
                   || $a->[$sortCol]->{number} <=> $b->[$sortCol]->{number}
                   || $a->[$sortCol]->{dateString}
                   cmp $b->[$sortCol]->{dateString}
@@ -1512,7 +1514,7 @@ sub emitTable {
             && $currentSortDirection == $SORT_DIRECTION->{'DESCENDING'} )
         {
             @curTable = sort {
-                     $b->[$sortCol]->{sortText} cmp $a->[$sortCol]->{sortText}
+                     NFKD($b->[$sortCol]->{sortText}) cmp NFKD($a->[$sortCol]->{sortText})
                   || $b->[$sortCol]->{number} <=> $a->[$sortCol]->{number}
                   || $b->[$sortCol]->{dateString}
                   cmp $a->[$sortCol]->{dateString}

-- GeorgeClark - 25 Jun 2015

See discussion In the IRC Logs

-- GeorgeClark - 25 Jun 2015

Fixed in EditRowPlugin, so removed from the "Applies To"

-- Main.CrawfordCurrie - 28 Sep 2015 - 08:49

Commenting is disabled while not logged in