Feature Proposal: Need better mechanism than {PluginsOrder} to control plugin execution order

Motivation

Sometimes plugins have dependencies on other plugins, or need to be executed last. individual handlers may have different constraints.

Description and Documentation

Requirements:

Need to be able to express the following:

1. Plugin A handler H needs to be executed before Plugin B handler H
2. Plugin A handler H needs to be executed after Plugin B handler H
3. Plugin A handler H needs to be executed before Plugin B handler H and again after Plugin B handler H

Perhaps:

1. Plugin A handler H needs to be executed after context identifier xyz is set
2. Plugin A handler H needs to be executed as early as possible
3. Plugin A handler H needs to be executed as late as possible

"handler H" could be black (all handlers) or a specific handler.

So, like zone dependencies, the plugins order is an acyclic graph.

Note that this execution order will only apply to handlers (like commonTagsHandler, beforeEditHandler). Registered tags already have a well defined execution order (left-right-inside-out).

Approach 1

As described in PluginOrderShouldSpecifyLastPlugins. Split the cfg{PluginOrder} variable into a First and Last section. Current setting is list of plugins, comma separated. Break the list into FirstPlugins;LastPlugins, delimited by the semi-colon. If the ;Last section is present, those plugins will be run after the alphabetical processing of un-named plugins.

Example:

AnnotateAttachmentPlugin,SpreadSheetPlugin;DBCachePlugin,SpreadSheetPlugin

Approach 2

Finer control over the relationships between specific plugins using a more complex syntax to express them.

constraints    ::= constraint ( ',' constraints )? ;
constraint     ::= ( 'REPEAT' )? handler constraintname ( plugin | '*' ) ;
handler        ::= plugin ( '::' handlername )? ;
constraintname ::= 'BEFORE' | 'AFTER' ;
plugin         ::= <name> ;
handlername    ::= <name> ;

Thus:

CommentPlugin::commonTagsHandler BEFORE WorkflowPlugin, 
   SpreadSheetPlugin BEFORE *,
   AnnotateAttachmentPlugin::beforeSaveHandler BEFORE *,
   REPEAT SpreadSheetPlugin::commonTagsHandler AFTER *

Note also that Approach 1's AnnotateAttachmentPlugin,SpreadSheetPlugin;DBCachePlugin,SpreadSheetPlugin is:

AnnotateAttachmentPlugin BEFORE *,SpreadSheetPlugin BEFORE *;DBCachePlugin AFTER *,REPEAT SpreadSheetPlugin AFTER *

Approach 3

Finer control as in (2), but removing the need for admins to maintain, or even know about our mess:

Each Plugin would contain its constraints within its shipped Config.spec, and by default they would mix together just as desired.

eg.

$Foswiki::cfg{Plugins}{Order}{CommentPlugin::commonTagsHandler}{BEFORE} = 'WorkflowPlugin';
$Foswiki::cfg{Plugins}{Order}{SpreadSheetPlugin::*}{BEFORE} = '*';
$Foswiki::cfg{Plugins}{Order}{AnnotateAttachmentPlugin::beforeSaveHandler}{BEFORE} = '*';
$Foswiki::cfg{Plugins}{Order}{SpreadSheetPlugin::commonTagsHandler}{REPEAT AFTER} = '*';

Note also that Approach 1's AnnotateAttachmentPlugin,SpreadSheetPlugin;DBCachePlugin,SpreadSheetPlugin is:

$Foswiki::cfg{Plugins}{Order}{AnnotateAttachmentPlugin}:BEFORE} = '*';
$Foswiki::cfg{Plugins}{Order}{SpreadSheetPlugin}{BEFORE} = '*';
$Foswiki::cfg{Plugins}{Order}{DBCachePlugin}{AFTER} = '*';
$Foswiki::cfg{Plugins}{Order}{SpreadSheetPlugin}{REPEAT AFTER} = '*';

-- Contributors: GeorgeClark - 3 Jul 2010, CrawfordCurrie - 18 Jul 2010, -- SvenDowideit - 19 Jul 2010

Discussion

See also Tasks.Item1580 and PluginOrderShouldSpecifyLastPlugins. It proposes a simple change to the {PluginsOrder} to allow specification of the "Last to execute" as well as the first to execute plugins.

Are you proposing that the user needs to configure the the plugin / handler order using a syntax shown in your verbatim blocks above? Would the typical admin have a chance of understanding how to do this? I'm concerned that it's too complicated.

-- GeorgeClark - 18 Jul 2010

For Tasks.Item1943, it would be nice if there was a way the InterwikiPlugin could execute a commonTagsHandler before the expansion of registered macros.

-- AndrewJones - 18 Jul 2010

George, sorry for missing your PluginOrderShouldSpecifyLastPlugins proposal - I didn't see it at the time, and didn't find it when searching. What you describe is of course a way to express a subset of the possible relationships.

Plugins execution order bites down hard with a specific group of plugins; SpreadSheetPlugin, TablePlugin, CommentPlugin, EditTablePlugin, ActionTrackerPlugin, DBCachePlugin, FormQueryPlugin - and just about anything else that uses a commonTagsHandler to process topic content. We have been eliminating it steadily with the introduction of registered tag handlers, so that the fine control is only required in a small number of cases. I'd prefer for plugins to themselves declare their dependencies, but to do that the plugins have to know about each other, which is impractical (many plugins are private to users).

In some cases - I'm thinking specifically of SpreadSheetPlugin and DBCachePlugin - are very delicate and difficult to get right. The first-last order just doesn't cut it, which was the reason for proposing the more complex relationships.

Andrew, the beforeCommonTagsHandler and commonTagsHandler are both called immediately before any macros are expanded. In effect, if plugin A declares beforeCommonTagsHandler and plugin B declares commonTagsHandler, this is equivalent to an execution order of A::commonTagsHandler BEFORE *, allowing A::beforeCommonTagsHandler to be moved to A::commonTagsHandler i.e. aspects of the execution order control are already hard-coded into the handler names (albeit rather inflexibly).

-- CrawfordCurrie - 19 Jul 2010

i prefer the dependency like approach in (2) more, but suggest Approach 3, as it can be implemented in a way that should require the installer not to know about it, or change anything unless there's a surprising wrinkle.

by this, I mean that rather than it being an ordered list, or string that has to be manually edited, it can be a hash of settings that each Plugin's Config.spec can set

eg.

$Foswiki::cfg{Plugins}{Order}{CommentPlugin::commonTagsHandler}{BEFORE} = 'WorkflowPlugin';
$Foswiki::cfg{Plugins}{Order}{SpreadSheetPlugin::*}{BEFORE} = '*';
$Foswiki::cfg{Plugins}{Order}{AnnotateAttachmentPlugin::beforeSaveHandler}{BEFORE} = '*';
$Foswiki::cfg{Plugins}{Order}{SpreadSheetPlugin::commonTagsHandler}{REPEAT AFTER} = '*';

where we use * as wildcard, and consider REPEAT as a modifier for BEFORE and AFTER.

the details of the example are only a first stab - the point is more that we should strive to make an already parsed, normalised form, so that foswiki admins don't need to make changes most of the time.

Note: in either Approach 2 or 3, a Checker would need to be implemented to help admins resolve, fix and disambiguate (and report) issues with the constraints

-- SvenDowideit - 19 Jul 2010

OK. Well, that's pretty much just a flattening out of the syntax I proposed, which I don't have a problem with. It might be cleaner to take a slightly different approachj:

$Foswiki::cfg{Plugins}{CommentPlugin}{Order}{commonTagsHandler}{BEFORE} = 'WorkflowPlugin';
$Foswiki::cfg{Plugins}{AnnotateAttachmentPlugin}{Order}{beforeSaveHandler}{BEFORE} = '*';
$Foswiki::cfg{Plugins}{SpreadSheetPlugin}{Order}{'*'}{BEFORE} = '*';
$Foswiki::cfg{Plugins}{SpreadSheetPlugin}{Order}{commonTagsHandler}{AFTER} = '*';

So the {Order} is in the same place as the {Module} and {Enabled}. REPEAT can be eliminated as BEFORE and AFTER are relative to another handler call (or set of handler calls), so we can have as many BEFORE and AFTERs as we want, as long as the dependency graph remains acyclic.

-- CrawfordCurrie - 19 Jul 2010

I like the syntax you proposed in your last comment, Crawford.

In any case, tinkering with plugins order - no matter how sofisticated the means are to specify it - is calling for trouble and very delicate to get right. Worst case would be that some TML works with one plugin order while other TML requires a different order on the same site. So that's not solvable using any fixed plugins order. I just mention it as this all is trying to lower the symptoms rather than fixing SpreadSheetPlugin (or TablePlugin or EditTablePlugin just to name another commonTagsHandler affine plugin).

So getting away from having a commonTagsHandler in these plugins is most probably better than any plugin order (by rationalizing parsing tables rather than reparsing tables again and again in separate plugins).

There are some handlers other than commonTagsHandler, i.e. those around saving a topic, that are less obvious when they get it wrong and thus even harder to fix. These plugins may tinker with the meta object about to be saved. While meta is passed around it might be enriched with additional data or analyzed. Obviously, a different plugins order will unveil different results. From the docu alone it might not be obvious what happens to meta during save. Nor is it obvious or foreseeable who should specify which handler comes first: in plugin A's Config.spec or in plugin B's?

So this proposal might most probably not fix this general problem.

Well, I'd welcome any means to be able to tinker with plugins order as a last resort . But be assured that getting it right in a specific constellation of plugins is very hard and might need a detailed code audit of the involved plugins ... a worst case of its own.

-- MichaelDaum - 20 Jul 2010

Note that if you want to pre-declare a plugins order in a Config.spec you can do it:

# **PERL**
# Define where handlers in this plugin need to be called, relative to the handlers in other plugins
$Foswiki::cfg{Plugins}{SpreadSheetPlugin}{Order} = {
   '*' => { BEFORE => '*' },
   commonTagsHandler => { AFTER => '*' }
};

-- CrawfordCurrie - 20 Jul 2010

What should happen if I have two plugins, each of which specify in their Config.spec that their handlers should be called first:

# **PERL**
# Define where handlers in this plugin need to be called, relative to the handlers in other plugins
$Foswiki::cfg{Plugins}{PushToTheFrontOfTheQueuePlugin}{Order} = {
   '*' => { BEFORE => '*' },
};

and

# **PERL**
# Define where handlers in this plugin need to be called, relative to the handlers in other plugins
$Foswiki::cfg{Plugins}{ThereShallBeNoOtherPluginsBeforeMePlugin}{Order} = {
   '*' => { BEFORE => '*' },
};

-- MichaelTempest - 20 Jul 2010

The ultimate sanction - alphanumeric ordering (alphabetti spaghetti)

-- CrawfordCurrie - 20 Jul 2010

like i said, a Checker would need to be written to help the admin resolve issues like clashes - and more importantly, impossible rules. You don't get to add this kind of complexity without writing helpers to fix the corners.

-- SvenDowideit - 20 Jul 2010

If I (1) install two plugins and then (2) enable the two plugins at once, and then (3) don't check for warnings in configure, I could be running my wiki with impossible rules. Sure, (1-2-3) is bad practice, but it happens. Foswiki must do something about clashes and impossible rules - infinite loops and "die" should be avoided :). That may be as simple as disabling one of the plugins and reporting the error on InstalledPlugins.

InstalledPlugins should also report the actual plugin order for each handler.

-- MichaelTempest - 22 Jul 2010