Item11697: Some wiki apps suffer because $topicObjects are either fully loaded or not (can't just fetch a part of a topic)

pencil
Priority: Enhancement
Current State: Proposal Required
Released In: n/a
Target Release: n/a
Applies To: Extension
Component: MongoDBPlugin
Branches: trunk
Reported By: PaulHarvey
Waiting For:
Last Change By: PaulHarvey
This is a continuation of Item11088, and isn't necessarily a SEARCH problem but more one reflecting the fact that MongoDBPlugin either fully loads a topic (resulting in transmisison of the entire JSON object from the DB over the wire), or it doesn't.

This task should focus on the problem of rendering bits (via $formfield(Foo) or VarQUERY) from hundreds -> thousands topics in a single page view.

In our case, the http://wiki.trin.org.au/Mangroves/Avicennia_marina page (and others) fall under this category where we render a $formfield() or two from a few hundred (maybe 200) different topics in a single page view.

This results in the page view sucking down 100s MiB from mongo, which takes several seconds (sometimes 10+).

If MongoDBPlugin could be instructed that we want a predefined list of meta parts from each hit in the result set, we could drastically improve performance and really slash the amount of data transmitted.

-- PaulHarvey - 27 Mar 2012

I agree with the principle behind this.

It has struck me quite a lot with my VDBI work how daft it is to:
  • Not load the required data while your doing the query, instead of performing another to load the selected data
  • Or load all the data when you only need some of it

But doesn't this hit a lot of core code? Wouldn't each topicObject need to track each META field as loaded or not?

Long and short of this is that it really requires a feature proposal as it needs core support.

-- JulianLevens - 26 Sep 2012

In brief discussions with SvenDowideit, it seemed t it might be possible to enhance the resultset iterator somehow (waving hands) in a way that query/store engines could dynamically/incrementally make each hit prefetch targeted topic parts, accumulating new parts to prefetch on the fly... or something...

yeah, it's an incredibly deep topic, and certainly needs more than just a task item.

-- PaulHarvey - 26 Sep 2012
 

ItemTemplate edit

Summary Some wiki apps suffer because $topicObjects are either fully loaded or not (can't just fetch a part of a topic)
ReportedBy PaulHarvey
Codebase trunk
SVN Range
AppliesTo Extension
Component MongoDBPlugin
Priority Enhancement
CurrentState Proposal Required
WaitingFor
Checkins AntiWikiSpamPlugin:876ba48cb0fc
TargetRelease n/a
ReleasedIn n/a
CheckinsOnBranches trunk
trunkCheckins AntiWikiSpamPlugin:876ba48cb0fc
Release01x01Checkins
Topic revision: r6 - 26 Sep 2012, PaulHarvey - This page was cached on 20 Oct 2020 - 10:56.

The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy