Item10461: SEARCH Speedup: ACL checks (and Paging) using iterator filters that can be optimized by Store implemetations.

see RefactorACLCheckOnResultSetAsFilter.

While this is initially implemented to improve MongoDBPlugin speed, the interface changes in the foswiki core will allow us to make dramatic improvements for all store & cache systems.

The core feature will move the ACL check from outside the ResultSet iteration to a FilterIterator wrapping the ResultSet as it comes back from the Search and query Algo. This FilterItr will be pluggable, to allow for other implementations, and for advanced query backends (like Mongo and Solr) can be a no-op - allowing full speed results.

The same technique will be applied to paging.

-- SvenDowideit - 17 May 2011

---

We've successfully made MongoDBPlugin massively faster at getting the first N hits of a query, but this doesn't help practical dashboards that need to show a paged view, and total number of hits for a given query.

Foswiki has no way of getting a "real" hit count unless it processes all 100,000 matching topics for ACLs. Because maybe the user only has permission to view 99,999 of them. Or just 6. Either way, we're back to square 1 with performance until we avoid forcing Foswiki to eval every topic for inclusion according to ACLs.

-- PaulHarvey - 09 Mar 2011

Raised to normal, because eventually I really want to Foswiki to only be processing the topics it absolutely needs to render a given page

-- PaulHarvey - 08 Apr 2011

this is going to be extremely difficult and require some very deep and risky changes to the core.

the result will be that the query in Item10432 should be very fast - at this point, line 745 of Foswiki::Search prevents it, because like most of the foswiki codebase, its coded to assume you need to parse and load a meta object to find out if its viewable.

-- SvenDowideit - 28 Apr 2011

If there's any way I can help, let me know.

-- PaulHarvey - 28 Apr 2011

because of the way they are interwoven, I've started by extracting the skipping for the pager - and trunk's pager should be O(1) as soon as i dcommit - as the core returns an aggregate iterator of ListIterators, and arrays are easy to dynamically skip 10,000 elements instantly.

-- SvenDowideit - 01 Jun 2011

this is feature complete - bugs should be raised in a new task.

unhappily, I commited MongoDBPlugin work under this task too, but

It needs a better headline, as it really should be a HUGE speedup - page 1000 should take about the same amount of time to render as page 1.

-- SvenDowideit - 01 Mar 2012

Commenting is disabled while not logged in