Feature Proposal: Support deferred execution or rendering of complex topics

Motivation

Some extensions have extended execution times that result in slow response to users, and excessive server load. Also some pages use very complex search and recursive search macros that can run for excessive time.

Also, some extensions fork to run other external processors like htmldoc, dot, topdf, etc. It would be good to move some of this off the web server directly and avoid the fork overhead especially if mod_perl is used

Description and Documentation

Captured a discussion from IRC

Examples

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: GeorgeClark - 21 Mar 2009

Discussion

(08:11:29 PM) gac410: Is there some way to design plugin / Sandbox usage to avoid forking for mod_perl's benefit?   
(08:12:12 PM) gac410: Wondering if DirectedGraphPlugin  (calls "dot" command)  or GenPDFAddOn (calls htmldoc)  should do something differently,  or just live with it?
(08:12:22 PM) SvenDowideit: well
(08:12:25 PM) SvenDowideit: _if_
(08:12:29 PM) SvenDowideit: you can batch things
(08:12:40 PM) SvenDowideit: so the output is only generated rarely
(08:13:00 PM) SvenDowideit: then you (and we need this kind of infrastructure) could make a 'work queue'
(08:13:13 PM) SvenDowideit: and have those things run be non-web-server
(08:13:22 PM) SvenDowideit: ala at/cron/else
(08:13:49 PM) SvenDowideit: a few weeks ago i was looking at using 'at' and the CPAN module for that kind of thing
(08:14:21 PM) SvenDowideit: but i needed it on windows, and of course, you can't use windows 'at' because you need a apssword or a gui or some stuffup
(08:14:29 PM) gac410: Interesting.   It would also take workload off the server - htmldoc and dot can be slow.
(08:14:35 PM) SvenDowideit: zactly :)
(08:15:05 PM) gac410: But would loose interactivity - have to check back for results?   
(08:15:10 PM) SvenDowideit: i recon that making a plugin thing that can offload to 'at' etc would be fantastic :)
(08:15:16 PM) SvenDowideit: usual story
(08:15:22 PM) SvenDowideit: its async processing
(08:15:33 PM) SvenDowideit: so you check, by checking :)
(08:16:17 PM) SvenDowideit: the nice thing is, once we have such a plugin, we can use it to automagically create the webnotify and webstats jobs
(08:16:25 PM) SvenDowideit: which would please alot of people :)
(08:19:08 PM) gac410: Have to think about how to package up request - and what work is done in foreground, what gets deferred.   Maybe some sort of SOAP / XML request to background. 
(08:19:32 PM) SvenDowideit: mmm
(08:19:46 PM) SvenDowideit: sounds alot more complex than what i was thinking
(08:20:05 PM) SvenDowideit: toss data into working/
(08:20:27 PM) SvenDowideit: call at to schedule (ok, so thats shelling out to something small)
(08:20:35 PM) SvenDowideit: and then let it do its stuff
(08:20:49 PM) SvenDowideit: and some sort of contention detection
(08:21:35 PM) SvenDowideit: ie toss data into working/web.topic.action so taht the next time foswiki thinks it needs to do it, is can do -e and not try to kick off another
(08:21:49 PM) gac410: Some is simple - "dot" - pass input file.   But htmldoc - all image attachments have to be extracted, potentially recursive topic rendering, etc.   
(08:22:15 PM) SvenDowideit: with html doc
(08:22:22 PM) SvenDowideit: don't do that :)
(08:22:43 PM) SvenDowideit: instead, get at to call the restHandler from the command line
(08:23:03 PM) SvenDowideit: so all you're really doing is passing the URLPARAMs to at, and getting it to do the real work
(08:23:36 PM) gac410: Hmmm  so backend becomes fully foswiki capable and not just a shell execution function.
(08:23:39 PM) SvenDowideit: so the web req and the at req are the same, just that the batching recognises one as a punt
(08:23:43 PM) SvenDowideit: exactly
(08:24:00 PM) SvenDowideit: that way you're adding a ligtweight bacther
(08:24:18 PM) SvenDowideit: and extension devs have to go very little (or even nothing)
(08:24:43 PM) SvenDowideit: ie, the batcherPlugin could use earlyInit to detect somethign that is configured as a 'batch'
(08:25:05 PM) SvenDowideit: and to do the at call with serialised URLPARAM and return a view
(08:25:24 PM) SvenDowideit: or a 'please wait' :)
(08:25:39 PM) SvenDowideit: with js refresh
(08:25:53 PM) gac410: I should probably capture this discussion in a feature request.  
(08:26:42 PM) SvenDowideit: na :) just write the plugin, publish at 10 and ..... profit
(08:27:30 PM) gac410: Might be able to provide a way that certain "heavy" search pages like recursive searches, etc.  could request a deferred search rather than tying up an apache thread as well.
(08:29:11 PM) SvenDowideit: yup
(08:29:25 PM) SvenDowideit: though that goes more towards the other model
(08:29:38 PM) SvenDowideit: um...
(08:29:44 PM) SvenDowideit: yeah - map-reduce idea
(08:30:00 PM) SvenDowideit: and pre-cached result-sets
(08:30:04 PM) SvenDowideit: ok, you're right
(08:37:26 PM) SvenDowideit: mmmm, do we have the meri&me tag definition feature req
(08:37:43 PM) gac410: not familiar with that.
(08:37:54 PM) SvenDowideit: its old old old :)
(08:38:05 PM) SvenDowideit: an idea i needed in 2005 actually :/
(08:41:14 PM) gac410: Maybe a special SECTION that is deferred rendering and cached.   Put anything like dot, complex search, etc in a deferred section and it is dispatched for asynchronous rendering.   Optional cache, so searches might not be cached, but dot, results, etc. would be cached and only re-rendered if changed.
(08:42:01 PM) gac410: Simplifies DirectedGraphPlugin because it would not have to maintain it's own cache then - the section containing the <dot> macro is cached.
(08:42:08 PM) SvenDowideit: i would suggest not
(08:42:35 PM) SvenDowideit: its better for the cache/delayed processor to not require changes to topics to activate
(08:43:00 PM) SvenDowideit: ie, better to have the batchPlugin have a cfg that allows the admin to identify what should be done offline
(08:43:13 PM) SvenDowideit: either by tag, regex, web, topic, etc
(08:44:06 PM) SvenDowideit: there are several techinques that you can leverage for it
(08:44:24 PM) gac410: So if the entire topic (and included topics) matches then defer the entire topic.   Could be complex to decide if an include would result in an expensive op that needs to be batched.
(08:44:39 PM) SvenDowideit: na, not regex the topic
(08:44:53 PM) SvenDowideit: basically, you can re-register a tag handler
(08:45:08 PM) SvenDowideit: over-riding the %DOT{}% handler with the defered one
(08:45:17 PM) SvenDowideit: or in htmldoc case, just detect the url
(08:45:24 PM) SvenDowideit: that sort of gubbins
(08:45:53 PM) gac410: I understand. I think  :-)  
(08:46:30 PM) SvenDowideit: the end point would also alow autodetection of 'this topic takes a long time to render' so cache&batch it
(08:46:41 PM) gac410: But %DOT{} is a bad example.   it is <dot>  multi-line structured input </dot>   that doesn't fit in a simple macro concept.
(08:46:50 PM) SvenDowideit: ah :)
(08:46:55 PM) SvenDowideit: bums
(08:47:07 PM) SvenDowideit: better re-write that
(08:47:32 PM) SvenDowideit: to use STARTSECTION ENDSECTION and then a %DOT{}% to process a named section :}
(08:47:50 PM) SvenDowideit: (ok, i don't have a quick answer, but i'll bet you find a way)
(08:48:05 PM) SvenDowideit: well there we go
(08:48:16 PM) SvenDowideit: the core _does_ have a multi-line IF
(08:48:26 PM) gac410: Well - I'm big on talking and rather poor on execution,  so don't hold breath!
(08:48:43 PM) SvenDowideit:  IF{"some condition" then="INCLUDE{some section}" else="INCLUDE{some other section}")
(08:48:57 PM) SvenDowideit: gac410, in that case, make the feature req
(08:49:07 PM) SvenDowideit: cos this discussion has solidified some of my thinking too :)
(08:49:12 PM) SvenDowideit: well worth it

---

Changing to parked proposal.

b-- Main.GeorgeClark - 13 Feb 2016 - 19:24