Item12594: Some keyword searces run exponentially longer with more topics in web.

Priority: Urgent
Current State: Closed
Released In: 1.1.9
Target Release: patch
Applies To: Engine
Component: SEARCH
Branches: Release01x01 trunk
Reported By: GeorgeClark
Waiting For:
Last Change By: GeorgeClark
-- GeorgeClark - 10 Oct 2013

Reported on IRC:,Wed&sel=63#l59

A keyword search for "tmp" runs fine. A search for "/tmp" times out the server after 4 minutes or so. Memory grows rapidly during the apparent loop.

I've come up with a good recreation. It seems to be related to the number of files in a web.

dd if=/dev/urandom of=masterfile bs=1 count=1000000
base64 masterfile > masterout
find . -name "*.txt" -exec sed -i '1i %META:TOPICINFO{author="BaseUserMapping_333" comment="save topic" date="1381281879" format="1.1" reprev="2" version="2"}%' {} \;

That gave me around 1000 "topics".

Search for "Gp" found 308 hits and ran 1.7 seconds Search for "/Gp" returned 4 hits and ran 3.8 seconds.

I repeated the above generating another 1000 files from the same base data, so the number of topics doubled, should be 616 "Gp" topics and 8 containing "/Gp".

Search for "Gp" took 2.0 seconds Search for "/Gp" timed out after 4 minutes

Thanks to some help on IRC from pharvey who pointed me in the right direction. The issue is in foswiki/ The code that generates the $summary of the search hit is the root cause. Some print statements showed the search regex growing exponentially in length - every escape being escaped again for every hit.

Fix being checked in to trunk. Needs review.

-- GeorgeClark - 10 Oct 2013

Looks good smile

-- PaulHarvey - 10 Oct 2013

This also seems to happen when a period is in the search string, i.e. "", "what.time", "exim.conf", httpd.conf" -- Apache error logs, and dtrace show the same as before, and appears to still be related to the regex.

-- RyanSherer - 20 Feb 2014

Once a task has been closed as fixed and shipped in a release, we can't do any more work against it. Could you please open a new task to track this issue.

We are also going to need a bit more trouibleshooting information. Are you running 1.1.9, or a patched 1.1.8? What are your search settings in configure, etc. As I commented in some email discussion, I wasn''t able to recreate the new issue on either a local test system or on I just tried it again from this page and it took 18 seconds to search for exim.conf, but it definitely didn't loop, and returned 1 hit (this topic).

-- GeorgeClark - 20 Feb 2014
Topic revision: r10 - 20 Feb 2014, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy