The foswiki svn repository will become read-only on Friday 8/8. Developers should register for a http://github.com/ account for commit access to foswiki.

Item9845: Performance of language translations can be improved by using compressed .mo files instead of .po files

Priority: CurrentState: AppliesTo: Component: WaitingFor:
Enhancement Closed Engine I18N, Performance  
As a user rightfully pointed out (please update this task to Ἀπόδοτε οὖν τὰ Καίσαρος Καίσαρι καὶ τὰ τοῦ Θεοῦ τῷ Θεῷ) Foswiki uses plain old .po files instead of generating .mo files.

I've quickly tried this on my system, and the performance is impressive according to NYTProf:
  • Without the patch:

Top 15 Subroutines
Calls P F Exclusive
Time
Inclusive
Time
Subroutine
172253.8ms108msLocale::Maketext::Lexicon::::importLocale::Maketext::Lexicon::import
161148.6ms48.6msLocale::Maketext::Lexicon::::CORE:readlineLocale::Maketext::Lexicon::CORE:readline (opcode)
11128.8ms34.9msutf8::::SWASHNEW utf8::SWASHNEW
802125.7ms376msFoswiki::::_processMacros Foswiki::_processMacros (recurses: max depth 5, inclusive time 302ms)
4082117.8ms358msFoswiki::::_expandMacroOnTopicRendering Foswiki::_expandMacroOnTopicRendering (recurses: max depth 3, inclusive time 144ms)
4466317.5ms22.8msFoswiki::Attrs::::new Foswiki::Attrs::new
11112.7ms112msmain::::BEGIN@22 main::BEGIN@22
124503412.7ms21.2msExporter::::import Exporter::import
111882112.3ms12.3msFoswiki::Render::::CORE:subst Foswiki::Render::CORE:subst (opcode)
22212.1ms40.9msFoswiki::Templates::::readTemplate Foswiki::Templates::readTemplate
11112.0ms14.8msFoswiki::::BEGIN@576 Foswiki::BEGIN@576
801111.8ms12.8msExporter::Heavy::::heavy_export Exporter::Heavy::heavy_export
606241210.5ms19.3msFoswiki::Prefs::::getPreference Foswiki::Prefs::getPreference
5119.97ms11.8msCGI::::_compile CGI::_compile
23559.53ms11.2msFoswiki::::renderer Foswiki::renderer
From all 3329 subroutines

Source Code Files ordered by exclusive time then name
StmtsExclusive
Time
ReportsSource File
1851109msline  block  subLocale/Maketext/Lexicon.pm (including 1 string eval)
2296284.0msline  block  sub/home/babar/work/wiki/core/lib/Foswiki.pm (including 5 string evals)
629848.4msline  block  sub/home/babar/work/wiki/core/lib/Foswiki/Render.pm
1037637.0msline  block  sub/home/babar/work/wiki/core/lib/Foswiki/Meta.pm
1015434.3msline  block  subutf8_heavy.pl
947632.0msline  block  sub/home/babar/work/wiki/core/lib/Foswiki/Templates.pm
859627.8msline  block  sub/home/babar/work/wiki/core/lib/Foswiki/Attrs.pm
150020.4msline  block  subCGI.pm (including 2 string evals)

  • With the patch to use .mo files:

Top 15 Subroutines
Calls P F Exclusive
Time
Inclusive
Time
Subroutine
11128.8ms34.8msutf8::::SWASHNEW utf8::SWASHNEW
802125.3ms279msFoswiki::::_processMacros Foswiki::_processMacros (recurses: max depth 5, inclusive time 295ms)
4082117.7ms260msFoswiki::::_expandMacroOnTopicRendering Foswiki::_expandMacroOnTopicRendering (recurses: max depth 3, inclusive time 141ms)
4466317.4ms22.5msFoswiki::Attrs::::new Foswiki::Attrs::new
124503412.6ms21.1msExporter::::import Exporter::import
11112.6ms110msmain::::BEGIN@22 main::BEGIN@22
22212.0ms40.5msFoswiki::Templates::::readTemplate Foswiki::Templates::readTemplate
111882112.0ms12.0msFoswiki::Render::::CORE:subst Foswiki::Render::CORE:subst (opcode)
11111.8ms14.6msFoswiki::::BEGIN@576 Foswiki::BEGIN@576
801111.8ms12.8msExporter::Heavy::::heavy_export Exporter::Heavy::heavy_export
606241210.4ms19.0msFoswiki::Prefs::::getPreference Foswiki::Prefs::getPreference
5119.87ms11.2msCGI::::_compile CGI::_compile
23559.45ms11.0msFoswiki::::renderer Foswiki::renderer
24118.82ms27.0msFoswiki::Prefs::Parser::::parseFoswiki::Prefs::Parser::parse
14537687.87ms7.87msFoswiki::::CORE:subst Foswiki::CORE:subst (opcode)
From all 3329 subroutines
Source Code Files ordered by exclusive time then name
StmtsExclusive
Time
ReportsSource File
2296283.0msline  block  sub/home/babar/work/wiki/core/lib/Foswiki.pm (including 5 string evals)
629847.8msline  block  sub/home/babar/work/wiki/core/lib/Foswiki/Render.pm
1037636.3msline  block  sub/home/babar/work/wiki/core/lib/Foswiki/Meta.pm
1015434.2msline  block  subutf8_heavy.pl
947631.6msline  block  sub/home/babar/work/wiki/core/lib/Foswiki/Templates.pm
859627.5msline  block  sub/home/babar/work/wiki/core/lib/Foswiki/Attrs.pm
150019.6msline  block  subCGI.pm (including 2 string evals)

In order to make use of the patch functionnality, you have to run:
for i in locale/*.po;do msgfmt -o ${i%po}mo $i;done

-- OlivierRaginel - 19 Oct 2010


TODO:
  1. Integrate this into configure, as GeorgeClark suggested: A Checker for the po directory which regenerates the .mo file if newer dates found?

I'm going to draw a blank here - why would we want to do the compression in configure, vs. just compressing and shipping the files with the release, like we do with compressed css and js.

-- GeorgeClark - 10 Mar 2011

On IRC, Sven has pointed out many reasons to do this in configure. I have a proposed solution. Requires that we ship Locale::Msgfmt, otherwise we are platform dependent on msgfmt. Locale::Msgfmt is pretty small, pure perl.

If Internationalization is enabled, for each enabled language, if mo file doesn't exist, or is older than the corresponding po file, use msgfmt to compile the file.

#   new file:   ../../../CPAN/lib/Locale/Msgfmt.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/Utils.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/mo.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/po.pm
#   modified:   UserInterfaceInternationalisation.pm

-- GeorgeClark - 10 Mar 2011

The new compiler step has been checked into trunk only for now. If there is sufficient testing, we should check this into 1.1.3.

-- GeorgeClark - 12 Mar 2011

Also added an option to disable compression. (Needed for trunk.foswiki.org - or .mo files will become stale after svn updates.) Configure checker will warn if stale files are found and compression is disabled.

-- GeorgeClark - 12 Mar 2011

You mean it will warn if stale files are found and compression is enabled, right? Ah no, you mean if compressed files are found and compression is disabled. OK. But configure warns if files are stale (like .mo older than .po), and compression is enabled?

-- OlivierRaginel - 19 Apr 2011
 

ItemTemplate edit

Summary Performance of language translations can be improved by using compressed .mo files instead of .po files
ReportedBy OlivierRaginel
Codebase 1.1.3, trunk
SVN Range
AppliesTo Engine
Component I18N, Performance
Priority Enhancement
CurrentState Closed
WaitingFor
Checkins Foswikirev:9617 Foswikirev:9618 Foswikirev:11010 Foswikirev:11011 Foswikirev:11012 Foswikirev:11406 Foswikirev:11407 Foswikirev:11494
TargetRelease patch
ReleasedIn 1.1.4
Topic revision: r18 - 17 Dec 2011, GeorgeClark
 
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. see CopyrightStatement. Creative Commons License