Feature Proposal: Add Paging To All Distributed Search Topics

Motivation

when running WebIndex on a too large web, we can DOS the server rendering (and searching for slower search algos)

Description and Documentation

I'd like to add a default pager, and set size to something that grep and render can handle - like 400)

perhaps default with a Web/Site pref?

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: SvenDowideit - 19 Apr 2011

Discussion

+1. Especially on slower VMs under load, something like 400 seems reasonable (add yet-another-pref? :-).

Also while you're at it, I guess you'll be removing use of VarTOPICLIST in WebIndex, which would "fix" Tasks.Item10653 I guess.

-- PaulHarvey - 25 Apr 2011

Disabling WebIndex all together has always been among the tweaks for xtra lage sites. I am pretty sure a WebTopicList is still a lot faster than a WebIndex. So I wonder why we need both.

-- MichaelDaum - 26 Apr 2011

the irony is that Tasks.Item10653 suggests that for really large sites, WebIndex is faster than WebTopicList - where large starts at 50,000 topics in a web and (in my test system, heading past 1,000,000 real topics (not generated, just collected over the years).

whatever happens, I'll be looking into that bug and the end result will probably be that I'll make SEARCH just as fast, and then define the TOPICLIST macro as a SEARCH.

If I can find big enough hardware (not blooming likely atm), I'll import a non-versioned backup of wikipedia into foswiki this year, and we can take it for a spin.

-- SvenDowideit - 26 Apr 2011

Careful: that report is on a mongodb setup exclusively. Assuming a standard search engine, any SEARCH might still be almost insane listing 50k topics .... which will DOS any browser and the user staring at the result anyway. That's why it is good to paginate things as suggested. Still, for all non-mongodb user, it might be good to use TOPICLIST as the fastest alternative to get the list of all topics in a web. If it isn't as fast as a SEARCH for a standard setup, then there's something else going wrong in TOPICLIST that needs fixing. The TOM interface that TOPICLIST uses is pretty straight forward as far as I can see, compared to SEARCH which is a hell lot more complicated.

-- MichaelDaum - 27 Apr 2011

agreed - Whatever (if anything) is done, it must be as fast or faster than the current option.

having marker and selection in SEARCH won't be a bad thing, and making SEARCH fast if format only contains $web and $topic makes alot of sense.

Interestingly, TOPICLIST appears to get most of its aleged speed from the fact it doesn't load the topic, and so doesn't obey ACL's to decide if it should render a topic. This should probably be doccoed.

wrt MongoDB - the reason its slower than SEARCH is because the Listeners don't get used for many Store API's including getWebList and eachTopic which I'll rectify soon.

-- SvenDowideit - 28 Apr 2011

I think this was merged and released a long time ago, marking it so.

-- SvenDowideit - 14 Mar 2012