Item9845: Performance of language translations can be improved by using compressed .mo files instead of .po files

Priority:	CurrentState:	AppliesTo:	Component:	WaitingFor:
Enhancement	Closed	Engine	I18N, Performance

As a user rightfully pointed out (please update this task to “Ἀπόδοτε οὖν τὰ Καίσαρος Καίσαρι καὶ τὰ τοῦ Θεοῦ τῷ Θεῷ”) Foswiki uses plain old .po files instead of generating .mo files.

I've quickly tried this on my system, and the performance is impressive according to NYTProf:

• Without the patch:

Top 15 Subroutines

Calls	P	F	Exclusive Time	Inclusive Time	Subroutine
17	2	2	53.8ms	108ms	Locale::Maketext::Lexicon::::importLocale::Maketext::Lexicon::import
16	1	1	48.6ms	48.6ms	Locale::Maketext::Lexicon::::CORE:readlineLocale::Maketext::Lexicon::CORE:readline (opcode)
1	1	1	28.8ms	34.9ms	utf8::::SWASHNEW utf8::SWASHNEW
80	2	1	25.7ms	376ms	Foswiki::::_processMacros Foswiki::_processMacros (recurses: max depth 5, inclusive time 302ms)
408	2	1	17.8ms	358ms	Foswiki::::_expandMacroOnTopicRendering Foswiki::_expandMacroOnTopicRendering (recurses: max depth 3, inclusive time 144ms)
446	6	3	17.5ms	22.8ms	Foswiki::Attrs::::new Foswiki::Attrs::new
1	1	1	12.7ms	112ms	main::::BEGIN@22 main::BEGIN@22
124	50	34	12.7ms	21.2ms	Exporter::::import Exporter::import
1118	82	1	12.3ms	12.3ms	Foswiki::Render::::CORE:subst Foswiki::Render::CORE:subst (opcode)
2	2	2	12.1ms	40.9ms	Foswiki::Templates::::readTemplate Foswiki::Templates::readTemplate
1	1	1	12.0ms	14.8ms	Foswiki::::BEGIN@576 Foswiki::BEGIN@576
80	1	1	11.8ms	12.8ms	Exporter::Heavy::::heavy_export Exporter::Heavy::heavy_export
606	24	12	10.5ms	19.3ms	Foswiki::Prefs::::getPreference Foswiki::Prefs::getPreference
5	1	1	9.97ms	11.8ms	CGI::::_compile CGI::_compile
23	5	5	9.53ms	11.2ms	Foswiki::::renderer Foswiki::renderer

From all 3329 subroutines

Source Code Files — ordered by exclusive time then name

Stmts	Exclusive Time	Reports	Source File
1851	109ms	line • block • sub	Locale/Maketext/Lexicon.pm (including 1 string eval)
22962	84.0ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki.pm (including 5 string evals)
6298	48.4ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki/Render.pm
10376	37.0ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki/Meta.pm
10154	34.3ms	line • block • sub	utf8_heavy.pl
9476	32.0ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki/Templates.pm
8596	27.8ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki/Attrs.pm
1500	20.4ms	line • block • sub	CGI.pm (including 2 string evals)

• With the patch to use .mo files:

Top 15 Subroutines

Calls	P	F	Exclusive Time	Inclusive Time	Subroutine
1	1	1	28.8ms	34.8ms	utf8::::SWASHNEW utf8::SWASHNEW
80	2	1	25.3ms	279ms	Foswiki::::_processMacros Foswiki::_processMacros (recurses: max depth 5, inclusive time 295ms)
408	2	1	17.7ms	260ms	Foswiki::::_expandMacroOnTopicRendering Foswiki::_expandMacroOnTopicRendering (recurses: max depth 3, inclusive time 141ms)
446	6	3	17.4ms	22.5ms	Foswiki::Attrs::::new Foswiki::Attrs::new
124	50	34	12.6ms	21.1ms	Exporter::::import Exporter::import
1	1	1	12.6ms	110ms	main::::BEGIN@22 main::BEGIN@22
2	2	2	12.0ms	40.5ms	Foswiki::Templates::::readTemplate Foswiki::Templates::readTemplate
1118	82	1	12.0ms	12.0ms	Foswiki::Render::::CORE:subst Foswiki::Render::CORE:subst (opcode)
1	1	1	11.8ms	14.6ms	Foswiki::::BEGIN@576 Foswiki::BEGIN@576
80	1	1	11.8ms	12.8ms	Exporter::Heavy::::heavy_export Exporter::Heavy::heavy_export
606	24	12	10.4ms	19.0ms	Foswiki::Prefs::::getPreference Foswiki::Prefs::getPreference
5	1	1	9.87ms	11.2ms	CGI::::_compile CGI::_compile
23	5	5	9.45ms	11.0ms	Foswiki::::renderer Foswiki::renderer
24	1	1	8.82ms	27.0ms	Foswiki::Prefs::Parser::::parseFoswiki::Prefs::Parser::parse
1453	76	8	7.87ms	7.87ms	Foswiki::::CORE:subst Foswiki::CORE:subst (opcode)

From all 3329 subroutines

Source Code Files — ordered by exclusive time then name

Stmts	Exclusive Time	Reports	Source File
22962	83.0ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki.pm (including 5 string evals)
6298	47.8ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki/Render.pm
10376	36.3ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki/Meta.pm
10154	34.2ms	line • block • sub	utf8_heavy.pl
9476	31.6ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki/Templates.pm
8596	27.5ms	line • block • sub	/home/babar/work/wiki/core/lib/Foswiki/Attrs.pm
1500	19.6ms	line • block • sub	CGI.pm (including 2 string evals)

In order to make use of the patch functionnality, you have to run:

for i in locale/*.po;do msgfmt -o ${i%po}mo $i;done

-- OlivierRaginel - 19 Oct 2010

---

TODO:

1. Integrate this into configure, as GeorgeClark suggested: A Checker for the po directory which regenerates the .mo file if newer dates found?

I'm going to draw a blank here - why would we want to do the compression in configure, vs. just compressing and shipping the files with the release, like we do with compressed css and js.

-- GeorgeClark - 10 Mar 2011

On IRC, Sven has pointed out many reasons to do this in configure. I have a proposed solution. Requires that we ship Locale::Msgfmt, otherwise we are platform dependent on msgfmt. Locale::Msgfmt is pretty small, pure perl.

If Internationalization is enabled, for each enabled language, if mo file doesn't exist, or is older than the corresponding po file, use msgfmt to compile the file.

#   new file:   ../../../CPAN/lib/Locale/Msgfmt.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/Utils.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/mo.pm
#   new file:   ../../../CPAN/lib/Locale/Msgfmt/po.pm
#   modified:   UserInterfaceInternationalisation.pm

-- GeorgeClark - 10 Mar 2011

The new compiler step has been checked into trunk only for now. If there is sufficient testing, we should check this into 1.1.3.

-- GeorgeClark - 12 Mar 2011

Also added an option to disable compression. (Needed for trunk.foswiki.org - or .mo files will become stale after svn updates.) Configure checker will warn if stale files are found and compression is disabled.

-- GeorgeClark - 12 Mar 2011

You mean it will warn if stale files are found and compression is enabled, right? Ah no, you mean if compressed files are found and compression is disabled. OK. But configure warns if files are stale (like .mo older than .po), and compression is enabled?

-- OlivierRaginel - 19 Apr 2011