Item13388: The %QUERY{...}% regex searches - fails on utf8 texts.

The %QUERY% fails when for any QuerySearch where the search is utf8 and

• the query contains a rexex with character class, such :upper: or the simple \w
• contains the ignore case modifier, e.g. (?i}utf8text
• or contains the lc uc for utf8

How to reproduce

Download the attachment. It is an raw topic file and contains a demo.

Tested on the trunk (commit 9fde37a000da7d8e1e3dfe4ab4cbe5871d09fec2).

Also, other errors

Because don't want spam the Tasks web with many utf8 related errors, the demo-topic contains other errors too, such

• QUERY type json fails for utf8,
• any CALC command from the spreadheet plugin manipulating utf8-texts fails
• search/regex fails
• EDITTABLE plugin causes some strange errors and so on...

-- JozefMojzis - 30 Apr 2015

I updated the topic to be ( i hope ) self-contained. (my test web has many testing topics and many testing variables). The previous version

• used some %VARIABLES% what i has defined in the WebPreferences - now changed
• and in the QUERY tried to query another topic's preferences value - now corrected to query itself.

So, GeorgeClark please download the topic and test it, not it should show: 15 errors. For ensure, i will attach the pdf version of the print... The page is intended to test clean distro testing (e.g. patternskin, etc..) so the Javascript could fail in other environments (i'm not an Javascript developer). It is tested with Mac's Safari and with Firefox/Firebug - and with fresh github clone. (Would be nice if someone could give some advices, how to make the JS part more universal.)

-- JozefMojzis - 04 May 2015

Ad comment from IRC

We'll probably document your Item13388 as a known issue.

Agree. IMHO we need release 1.2 :). Foswiki is now usable for editing UTF8 texts. Need document the issues:

• any %QUERY% or %SEARCH% will fail
  1. for regex searches containing character classes such \w or [[:upper:]] and such e.g. the case insensitive operator too (?i)
  2. for lc and uc case conversions
• the string functions of the SpreadSheetPlugin doesn't works with utf8
• the %EDITABLE% macro somewhat broke the utf8 topic text... (only some characters). Maybe this issue got fixed with the Item13391 - not tested yet.

-- JozefMojzis - 04 May 2015

The example given is working as expected, on unicode core. Closing.

-- CrawfordCurrie - 19 May 2015

Commenting is disabled while not logged in