You are here: Foswiki>Tasks Web>Item13566 (26 Sep 2016, MichaelDaum)Edit Attach

Item13566: Speed-aware reading encoded files

pencil
Priority: Enhancement
Current State: Confirmed
Released In: n/a
Target Release: n/a
Applies To: Engine
Component: Performance, Unicode
Branches:
Reported By: JozefMojzis
Waiting For:
Last Change By: MichaelDaum
Benchmarked different file-reading and decoding methods, and got some surprising results.

So asked an question in the Stackoverflow. http://stackoverflow.com/questions/31598938/benchmarking-utf8-file-read-explanation-of-the-differences

The answer pointed me to the module CPAN:Unicode::UTF8, it allows reading files 5 times faster as using Encode. The question contains some benchmarking code too.

Therefore, (and learned from the CGI-problem lesson too), IMHO Foswiki shouldn't use any CPAN modules directly, but thru its own interface modules, like Foswiki::Encode - instead of the plain perl's Encode:. Such approach could allow easily modify, replace the backend implementation, e.g. in this case the file-reading routine could be:

if( the_store_is_utf8 ) {
    read RAW file and use Unicode::UTF8 for the speed gain
} else {
    read the file using Encode
}

-- JozefMojzis - 27 Jul 2015

Setting this to confirmed. But does need more investigation.

-- GeorgeClark - 25 Sep 2016

Jozef, which perl version have you been benchmarking. Note that anything befor 5.22 is not worth it. I just looked at Unicode::UTF8's activity on github. ... which logs most recent commit back in 2013. Hm.

-- MichaelDaum - 26 Sep 2016
 

ItemTemplate edit

Summary Speed-aware reading encoded files
ReportedBy JozefMojzis
Codebase trunk
SVN Range
AppliesTo Engine
Component Performance, Unicode
Priority Enhancement
CurrentState Confirmed
WaitingFor
Checkins
TargetRelease n/a
ReleasedIn n/a
CheckinsOnBranches
trunkCheckins
masterCheckins
ItemBranchCheckins
Release02x01Checkins
Release02x00Checkins
Release01x01Checkins
Topic revision: r3 - 26 Sep 2016, MichaelDaum - This page was cached on 22 Nov 2020 - 16:25.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy