Item10461: SEARCH Speedup: ACL checks (and Paging) using iterator filters that can be optimized by Store implemetations.

pencil
Priority: Enhancement
Current State: Closed
Released In: 2.0.0
Target Release: major
Applies To: Engine
Component: SEARCH, MongoDBPlugin
Branches: master
Reported By: PaulHarvey
Waiting For:
Last Change By: GeorgeClark
see RefactorACLCheckOnResultSetAsFilter.

While this is initially implemented to improve MongoDBPlugin speed, the interface changes in the foswiki core will allow us to make dramatic improvements for all store & cache systems.

The core feature will move the ACL check from outside the ResultSet iteration to a FilterIterator wrapping the ResultSet as it comes back from the Search and query Algo. This FilterItr will be pluggable, to allow for other implementations, and for advanced query backends (like Mongo and Solr) can be a no-op - allowing full speed results.

The same technique will be applied to paging.

-- SvenDowideit - 17 May 2011


We've successfully made MongoDBPlugin massively faster at getting the first N hits of a query, but this doesn't help practical dashboards that need to show a paged view, and total number of hits for a given query.

Foswiki has no way of getting a "real" hit count unless it processes all 100,000 matching topics for ACLs. Because maybe the user only has permission to view 99,999 of them. Or just 6. Either way, we're back to square 1 with performance until we avoid forcing Foswiki to eval every topic for inclusion according to ACLs.

-- PaulHarvey - 09 Mar 2011

Raised to normal, because eventually I really want to Foswiki to only be processing the topics it absolutely needs to render a given page smile

-- PaulHarvey - 08 Apr 2011

this is going to be extremely difficult and require some very deep and risky changes to the core.

the result will be that the query in Item10432 should be very fast - at this point, line 745 of Foswiki::Search prevents it, because like most of the foswiki codebase, its coded to assume you need to parse and load a meta object to find out if its viewable.

-- SvenDowideit - 28 Apr 2011

If there's any way I can help, let me know.

-- PaulHarvey - 28 Apr 2011

because of the way they are interwoven, I've started by extracting the skipping for the pager - and trunk's pager should be O(1) as soon as i dcommit - as the core returns an aggregate iterator of ListIterators, and arrays are easy to dynamically skip 10,000 elements instantly.

-- SvenDowideit - 01 Jun 2011

this is feature complete - bugs should be raised in a new task.

unhappily, I commited MongoDBPlugin work under this task too, but

It needs a better headline, as it really should be a HUGE speedup - page 1000 should take about the same amount of time to render as page 1.

-- SvenDowideit - 01 Mar 2012

 
Topic revision: r33 - 05 Jul 2015, GeorgeClark - This page was cached on 10 Dec 2016 - 10:08.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License