Item11476: Improve RcsLite worst-case performance.

Item11454 reignited discussion about making RcsLite the default $Foswiki::cfg{Store}{Implementation}.

I've helped several users through debugging slow Foswiki installations, they're using mod_fcgid but are still using RcsWrap. Generally, switching to RcsLite means response times go from minimums of circa 2-3s down to less than 1s.

However, there are some concerns with RcsLite before we make it the default:

• Immediately, we could make a configure checker warn if we detect FastCGIEngineContrib or ModPerlEngineContrib is installed and RcsWrap is configured?
• Presently, loads the entire file into memory. Need to make it smarter - I suspect some Foswiki operations can be serviced by only processing the first few lines, or should we just refactor it to work on file streams generally?
• RcsLite is much slower when ,v files are quite large (dozens of MB). In this case the external RCS tool is faster (it's pure c, after all, so it will beat any pure perl in raw i/o throughput).
• Some users have reported problems on very large attachment histories, even with the 32-bit RCS tools (their ,v files exceed 2GB!) - so I think we need the ability to support a hybrid setup, RcsLite for data/ and RcsWrap for pub/

-- PaulHarvey - 25 Jan 2012

Putting all current concerns to the side with regards to worst case scenarios where RcsLite doesn't seem robust enuf yet.

Going down from 2-3 seconds to under 1 second is a massive speedup that users will immediately feel doing their everyday work using foswiki.

It should be clear that making RcsLite the default store implementation should have high priority.

Fixing RcsLite making it more rubust is a relatively simple task compared to all the other untertakings to improve speed. That's why I make this task Urgent, not Enhancement, as speed improvement is a must, not a nice-to-have.

-- MichaelDaum - 26 Jan 2012

Deferring this until 1.2. We don't ship fcgid or mod_perl support by default. Does this rise to the level of needing a feature proposal?

-- GeorgeClark - 16 Feb 2012

To be fair, RcsLite still helps even without mod_perl/fcgid, on pages where you're hitting a lot of topics (giant SEARCHes, for example).

This probably shouldn't block 1.1.5, but I'd hope we can still get this into Release01x01 at some point

-- PaulHarvey - 16 Feb 2012

I finally got round to performing a bunch of optimisations on this code, that have accelerated it quite a bit.

There's still a rather stupid problem with the code that the operation which most needs acceleration is getInfo, but RCS stores the "log" part of that with the text of the change, and not in the version history block at the head of the file. That forces you to read much more than is really needed. It's impossible to optimise further using the linear read approach.

I suspect I've gone as far with this code as it's sensible to go, so marking it for release.

-- CrawfordCurrie - 04 Apr 2013

There is a comment / SMELL in RcsLite:

# SMELL: This code uses the log field for the checkin comment. This field is alongside the actual text
# of the revision, and is not recorded in the history. This is a PITA because it means the comment field
# can't be retrieved without reading up to the text change for the version requested - even though foswiki
# doesn't actually use that part of the info record for anything much. We could rework the store API to
# separate the log info, but it would be a lot of work. Using this constant you can ignore the log info in
# getInfo calls. The tests will fail, but the core will run a lot faster.
use constant CAN_IGNORE_COMMENT => 0;    # 1

Should we document this, or maybe even run this way by default. If we can stop reading the rcs file before getting into the body of the diff, it would seem that would be a huge boost.

-- GeorgeClark - 17 Jun 2014

Commenting is disabled while not logged in