Feature Proposal: Process addToHEAD adds

Motivation

Currently plugins call TWiki::Func::addToHEAD to add javascript and css text and file includes to the html <head> tag. The TWiki core blindly adds these to the header.

If these 'adds' were processed, we could:

• use ordering/priorities for includes, for example to add things to the beginning instead of at the end
• remove duplicate calls to the same files
• accumulate javascript files to one file and dynamically compress it (or cache the compressed file) to improve page rendering speed
• the same for css files

Specification

%ADDTOHEAD%

You can write %ADDTOHEAD{...}% in a topic or template. This variable accepts the following parameters:

• _DEFAULT optional, id of the head block. Used to generate a comment in the output HTML.
• text optional, text to use for the head block. Mutually exclusive with topic.
• topic optional, full TWiki path name of a topic that contains the full text to use for the head block. Mutually exclusive with text.
• requires optional, comma-separated list of id's of other head blocks this one depends on.

%ADDTOHEAD% expands in-place to the empty string, unless there is an error in which case the variable expands to an error string.

TWiki::Func::AddToHEAD

TWiki::Func::addToHEAD is modified to accept two new parameters, $text and $requires, which have the same behaviours as the equivaent %ADDTOHEAD% parameters.

%RENDERHEAD%

%RENDERHEAD% should be written where you want the sorted head tags to be generated. This will normally be in a template. The variable expands to a sorted list of the head blocks added up to the point the RENDERHEAD variable is expanded. Each expanded head block is preceded by an HTML comment that records the ID of the head block.

Head blocks are sorted to satisfy all their requires constraints. The output order of blocks with no requires value is undefined. If cycles exist in the dependency order, the cycles will be broken but the resulting order of blocks in the cycle is undefined.

Examples

In templates / topics:

%ADDTOHEAD{"no_style" text="<style> .none {} </style>"}%
%ADDTOHEAD{text="<style> .chic {} </style>" requires="no_style"}%
%ADDTOHEAD{"doggy" topic="Styles.DoggyStyle" requires="no_style,hair_style"}%

in plugins / contribs:

TWiki::Func::addToHEAD("hair", "<style> .hair {} </style>");
TWiki::Func::addToHEAD("free", "<style> .free {} </style>", "doggy,hair");

-- Contributors: ArthurClemens, CrawfordCurrie, SvenDowideit, MichaelDaum

Discussion

Refactored discussion to date into the spec, and fleshed out the spec to deal with a range of ordering cases that I have encountered.

-- CrawfordCurrie - 29 Dec 2007

There have been a number of recent works in the JS world to add automatic dependancy and ordering to the loading of JS libraries. When I'm not no dialup I'll add a few links - though I'm sure that someone else has come across them too - see ajaxian.com (there's even one that uses TWiki... and the developer has built a very simple version of the TWiki IDE I've been building with Dojo)

-- SvenDowideit - 29 Dec 2007

Interesting reading. Managing dependencies between JS libs in JS is an excellent idea, and is certainly the wave of the future. If we could get agreement on what JS toolkit we were going to use, it would be credible (preferable) to use this as the only approach. My gut tells me, though, that we will have the solution described above much sooner than we have agreement on a JS toolkit (or even a common facade)

-- CrawfordCurrie - 30 Dec 2007

FWIW, i've been using a simple patch on one of my projects so that the HEAD sections are ordered by their id alphanumerically. Worked fine for the purpose (KISS), and adds a lto less performance overhead as what you describe here (i'm increasingly worried by the way too poor performance TWiki gives). Of course, it doesn't allow you to say before/after, you'd have to make sure the names are in the right order alpha-numerically (perhaps by using names like 000_foo and 200_bar and such).

-- KoenMartens - 01 Jan 2008

I obviously need to dig out the reference - the one I'm thinking of was specifically written to cover most of the existing toolkits, and avoids locking TWiki out of co-existing with other toolkits. For eg, it looks pretty naf when your site uses a mix of Dojo and YUI UI components.

-- SvenDowideit - 02 Jan 2008

Crawford has liften his concern. This leaves no ourstanding issues on this one. And we are way past the 14-days. So it is accepted by concensus.

-- KennethLavrsen - 02 Jan 2008

mmm, i only just noticed this was 'accepted' dispite my request that we use dependancies - also mentioned in MoveToJQuery.

-- SvenDowideit - 10 Jan 2008

It is always an interpretation on my part if the statements given have reached consensus. The sure way to signal concern to me is to add your name to the ConcernRaisedBy field. I do not remove a name unless I am very sure and I prefer people remove themselves.

But our process is flexible - like TWiki is. I choose now to flip the state to ReadyForReleaseMeeting. Then we can discuss it and see if we can make it move on there.

-- KennethLavrsen - 11 Jan 2008

Thinking on this proposal and its relationship to TWiki Apps, I think we also need an =%ADDTOHEAD{}% specifically to allow TWikiApps to request the addition of 'safe and installed' javascript only when its actually needed. There have been a number of times I have added a Plugin.pm to a contrib just to get some css or javascript added to the header.

-- SvenDowideit - 11 Jan 2008

"There have been a number of times I have added a Plugin.pm to a contrib just to get some css or javascript" - this is addressed by InitialiseNonPluginExtensions. Perhaps it would be appropriate to re-open that issue?

-- CrawfordCurrie - 11 Jan 2008

For my needs, I don't want Perl code to add to head, its much more than appropriate.

-- SvenDowideit - 11 Jan 2008

OK, but some of us do want perl code. This proposal exposes the perl code we need via %TMPL:HEAD, so the main point you are making here - correct me if I'm wrong - is that you want %ADDTOHEAD, which can be used in topics, rather than %TMPL:HEAD, which can only be used in templates. My main concern with that is security. %TMPL:HEAD requires write access to templates, whereas anyone can call %ADDTOHEAD in a topic - perfect for cross-scripting (it would defeat SafeWikiPlugin)

-- CrawfordCurrie - 04 Feb 2008

Discussed on GeorgetownReleaseMeeting2008x02x04. No decision. Need to be discussed on Codev more

-- KennethLavrsen - 04 Feb 2008

Yep. I want %ADDTOHEAD that works in topics, so that a TWikiApp could cause the JS to only be brought in when needed. If its restricted to TMPL's then we're not helping TWikiApp developers, and forcing JS to be loaded much more often than it would really be needed.

Allowing anyone to call %ADDTOHEAD from a topic, is not a problem - I generally use TWiki::Func::AddToHEAD to add things from templates in such a way as to leverage the SKIN setting. I propose that %TMPL:HEAD and %ADDTOHEAD do the same - reference a template name, that can have a skin path evaluated on it.

for example ==, would be resolved to either popup.dojo.tmpl or popup.jquery.tmpl, both of which would be able to attach to approporiately classed css targets.

-- SvenDowideit - 04 Feb 2008

On re-reading the proposal, I realised several BIG things.

1. so long as a user can use %TMPL::HEAD in any topic, no extra TML is needed
2. More significantly replace the new %TMPL::HEAD in the revamped proposal with %TMPL::DEF and you have a facility that we always needed, and would lead to seriously simplified skins - the ability to re-use the lower level definition

ie, There are places where it would simplify things to be able to write

%TMPL:DEF{Widget}%%TMPL:P{Widget}%Some extra stuff for the non-default skin version%TMPL:END%

obviously this is not possible today, but with the added syntax above, we can

%TMPL:DEF{"Widget" concatenate="on"}%Some extra stuff for the non-default skin version %TMPL:END%

I think it would be more powerful (and easier to read) to change the concatenate feature to allow recursive definitions, as we can then append and prepend, either through the implicit

%TMPL:DEF{Widget}%Some extra stuff for the %TMPL:P{"Widget"}% non-default skin version %TMPL:END%

or the explicit

%TMPL:DEF{Widget}%Some extra stuff for the %TMPL:P{"Widget" previous="on"}% non-default skin version %TMPL:END%

Similarly, it is likely that changing the TMPL:DEF syntax to allow depends and allowes (synonyms for before and after ) might make some odder ordering constructs (where over-riding skins want to add something to the middle of a set of TMPL:DEFs) simpler.

-- SvenDowideit - 31 Mar 2008

I explicitly excluded the possibility of using TMPL:HEAD in topics in the proposal above to maintain a clear separation between "compile time" and "run time". At the moment we already have %TMPL:P available in topics, which IMHO is a mistake, because it is syntactically identical to %TMPL:P in a template, but semantically very different. Access permissions (which look like TWiki variables but aren't) are another example where using the same syntax just causes confusion. So I'd prefer to use %ADDTOHEAD (and deprecate %TMPL:P in topics (replace it with %TEMPLATE{) while we're at it )

Recursive TMPL:DEF definition shouldn't require any additional syntax, just a code change to make it expand the contents of a new definition of %TMPL:P{X} before it redefines %TMPL:P{X}. IMHO this is a powerful and necessary concept and should be implemented.

Note that TMPL:DEF in templates is already used to create mixin-like behaviours, by defining base templates such that their component parts can be redefined. For example,

%<nop>TMPL:DEF{"TopBar"}% %<nop>TMPL:P{"After"}% ... %<nop>TMPL:P{"After"}% %<nop>TMPL:END%
....
%<nop>TMPL:DEF{"Before"}% ??? %TMPL:END%
%<nop>TMPL:DEF{"After"}% ??? %TMPL:END%

The biggest problem I see with the way this approach has been applied is the difficulty for someone reading the templates; in the example above, there is no namespace that associates "After" with "TopBar", which can make re-use a nightmare. Better use of namespaces could make the templates much easier to read, and much easier to mixin/subclass. For example,

%TMPL:DEF{"TopBar"}% %TMPL:P{"TopBar.After"}% ... %TMPL:P{"TopBar.After"}% %TMPL:END%
....
%TMPL:DEF{"TopBar.Before"}% ??? %TMPL:END%
%TMPL:DEF{"TopBar.After"}% ??? %TMPL:END%

This approach is already used in things like the attachment tables templates, and wouldn't be hard to apply onto the existing skins; it's just a question of discipline. If you parsed the names, you could even provide some implicit definitions, such as "Before" and "After":

%TMPL:DEF{"TopBar"}% ... %TMPL:END%
....
%TMPL:DEF{"TopBar.Before"}% ??? %TMPL:END%
%TMPL:DEF{"TopBar.After"}% ??? %TMPL:END%

But this is all getting off the track of this proposal. The debate about %ADDTOHEAD in topics is still open. I understand your arguments for it, and agree that the proposal should be extended to encompass it.

-- CrawfordCurrie - 31 Mar 2008

At the GeorgetownReleaseMeeting2008x04x14 it was decided that Crawford should be given the chance to update the proposal as he indicates he wants to do.

At next release meeting we will vote for whatever proposal is there at the top of this proposal.

-- KennethLavrsen - 14 Apr 2008

Note that a similar addToBottom for script files would be nice also for performance, according to http://developer.yahoo.com/performance/rules.html#js_bottom

-- ColasNahaboo - 14 Apr 2008

OK, I'll refactor it to try and take Sven's concern into account.

-- CrawfordCurrie - 15 Apr 2008

one thing I think we should consider:

Will SECTIONS that are brought in have their Set statements processed? Our current implementation of INCLUDE does not, so consistency would suggest no - but is this the best decision?

-- SvenDowideit - 15 Apr 2008

If I understand this right, STARTSECTIONS of type head that are burried in the topic text of a base topic will be added to the html head when this base topic is rendered. Does this mean, these head sections will have to be part of the topic text of this one topic to take effect? What if I don't want to mix this with my net data? I'd like to hide the additional javascript inside the depth of some twikiapp, adding it to the html head.

Why not simply leveraging the new TWiki::Func::addToHEAD() from perl-land to tml-land using an %ADDTOHEAD% tag? That would be much easier for app writers.

%ADDTOHEAD{prio="0-99999" key="foo" "<script ...</script>"}%
%ADDTOHEAD{topic="HeadShop" section="..."}%
%PREPENDTOHEAD{...}%
%APPENDTOHEAD{...}%
%RENDERHEAD{format="..."}%

• prio: numerical value to force insertion at a specific position
• key: allocate a key for the additon. keys are unique. subsequent additions with the same key will override.
• _DEFAULT: inline content to be added
• topic: content of a topic to be added
• section: used together with topic. see %INCLUDE
• PREPENDTOHEAD is the same as ADDTOPHEAD{prio="-infinit"}
• APPENDTOHEAD is the same as ADDTOPHEAD{prio="+infinit"}
• RENDERHEAD: reads out all registered head content in the given order, special variables $key, $prio, $text in the format string; format defaults to "$text"

-- MichaelDaum - 15 Apr 2008

Sven, yes, I personally think it is the best decision.

Michael, this is leveraging addToHead as a tag. Sections can be used in templates and topics, same as any other tag, However I wanted it to be a section tag for two reasons; first, to leverage it's existing unique position in the rendering cycle, and second to allow me to specify a block of content. I'm not hung up on this; if someone else can propose a better syntax that achieves the same thing, they are welcome to do so.

I'll repeat again; ordering using a simple integer positional modifier does not work. It is too limiting, especially when someone has already used the integer for the position you wanted - even if you can find it out (not so easy sometimes).

-- CrawfordCurrie - 15 Apr 2008

I assume these talk pages are an experiment. Personally I am more confused than helped by this splitting into two. I now have to edit the form in the proposal topic and the status updates in the discussion topic. I added a link to the original topic to ease navigating back but I still see this as a cheap way of trying to make the discussion feature from Mediawiki and I do not find that this enhances my life or makes things easier. The included proposal at the topic of this topic was only partly included adding to the confusion, so I removed the inclusion. I really do not want to see these talk pages on more proposal topics. It adds extra work for me.

It is extremely difficult for my brain to absorb the message of the proposal with all those bold words where the word bolded is not even itself important content. Please save those bolds to things that are really important. Use italic instead. It is less intrusive on the brain.

About the proposal.

Now that the proposal has been refactored I have the feeling the scope of the implementation has also been increased beyond what the proposer originally committed. This may not be a problem if the proposer agrees. If not then the proposer of the enhancements should chip in and add himself as committed co-developer. Otherwise we end up with a proposal which is accepted but never gets implemented.

-- KennethLavrsen - 15 Apr 2008

Replaced the link to the original topic with the Talk page View Template - The Talk and Topic pairs are intended to be symetrical, with action bar links between them. I also don't think its worth adding talk topics in a talk web like Codev, but Some people wanted to try it.

Of course, this TalkTalkTalk discussion should be moved to the TalkTalkTalk topic.

-- SvenDowideit - 16 Apr 2008

Actually, Kenneth, the only thing that has changed since the last version is the use of the SECTION syntax, which is more consistent with what is already there.

I think the people asking for alphanumeric ordering are really missing the point; that's not the hard bit.

-- CrawfordCurrie - 16 Apr 2008

Which point did I miss? Please explain.

-- MichaelDaum - 16 Apr 2008

I wrote a test implementation which I shared with Michael, Sven and Arthur. Michael gave me feedback, and we we discussed some key points:

Michael's concerns with my implementation:

• code efficiency (complexity)

My concerns with Michael's proposal:

• partial ordering
• multi-line input

My concerns with my implementaion:

• code efficiency (complexity)

We has a useful IRC discussion in which we thrashed out the following:

• Michael's solution is better. Use an ADDTOHEAD/RENDERHEAD tag approach.
• Must define ranges of positional IDs that can be used for different things (js, CSS etc) that is wide enough to allow insertion

I agreed to update the spec and the test implementation, and Michael agreed to define the ranges.

-- CrawfordCurrie - 18 Apr 2008

I can't see enough explaination to understand what looks like a backwards step to me -

This change means that Instead of having a plugin or contrib declaring an explicit 'dependancy' on a piece of head code (such as 'jQuery') I need to know a registered 'number' that implies 'jQuery' and use that (a know poor mechanism for dependancies).

And we've totally lost the ability to define Header elements within templates, as required by the parent FeatureRequest - MoveToJQuery.

I'm sorry I havn't had time to look at the plugin implementation, but I see this as a very backwards step, and one which is even less useful than simply sorting addToHEADs alphabetically (a 5 character patch to the current code.

-- SvenDowideit - 19 Apr 2008

The Code I'm writing works more like:

TWiki::Func::addToHEAD(
        active=>1,              #defaults to 1
        name=>'twisty', 
        needs=>'dojotoolkitcontrib',
        definition=>'text',
        text=>$text,            # which of the below parameters is used depends on the 'definition' parameter
        template=>'',           
        templatedef=>'',
        topic=>'',
        section=>'',
);
%<nop>ADDTOHEAD{
    name='twisty' 
    needs='dojotoolkitcontrib'
    definition='text'
    text='&lt;style&gt;....'
}%
(and a future proposal)
%<nop>TMPL:DEF{
    'twisty' 
    needs='dojotoolkitcontrib'
}% ... %<nop>TMPL:END%

• active = are we declaring a section that will definatly be used, or one that will only be used if there is a dependancy on it. (defaults to 1 or 'true')
• name is the name of the section that you are defining
• needs is a comma seperated list of HEAD sections that must come before this one
• definition declares where this name=d section is defined (can be =text, topic, template, section)
  • text - section is defined in the named text parameter
  • topic - section is defined as the entire topic named in the topic parameter
  • template - section is defined as the entire expanded template 'file' using the skin path (will call loadTemplate)
  • templatedef - section is defined as the entire expanded template def named (will assume template file is loaded)
  • section - uses the topic and section named parameters to define a section (equivalent to text='%INCLUDE{"" section=""}%'
  • toend - allows the %ADDTOHEAD marker to be paried with a %ADDTOHEADEND{name=''}% section (as in named section both must have the same name.

For backwards compatibility, AddToHEAD can still be called in the 4.2.0 fashion, resulting in a named HEAD section with no dependancies.

-- SvenDowideit - 28 Apr 2008

This gets too complicated and you get into more trouble than needed.

Your proposed syntax is quite complicated / massive.

For example, what about circular dependencies, which names are predefined/reserved, etc.

Where is the qualitative advantage of a dependency hierarchy versus a numerical approach because in the end any dependency hierarchy needs to be translated into an optimal linear ordering anyway ... which means it simply can't be more powerful than any explicit numerical approach.

Last but not least you will have to document and register well known names used in dependencies the same way numeric ranges have been proposed.

Guys, this proposal is rather dinky compared to what is ahead of us for TWiki-5.0. Let's pocket it using a simple solution, and then we need to move on.

-- MichaelDaum - 29 Apr 2008

I have attached the latest implementation of a plugin that mixes the best of the proposals to date to realise a partial ordering (via a requires relation) in linear time.

I've now made three implementation proposals, so I'm stepping back from this discussion.

-- CrawfordCurrie - 29 Apr 2008

Is the proposal topic ProcessAddToHeadAdds updated?

Is the original proposer willing to implement the new proposal?

I need a yes and yes (or yes and new committed developer) to bring this to release meeting.

I merged this proposal topic back to ONE topic because the discussion feature destroyed the history view making it impossible to see what people change in the proposal.

-- KennethLavrsen - 09 Jun 2008

Crawford's plugin works excellent. Do others see an urgent need to have pos next to requires? Otherwise I propose to move forward and implement this into the core.

-- ArthurClemens - 28 Jul 2008

Sven and Crawford. Do you agree that we can move ahead with this?

-- KennethLavrsen - 04 Aug 2008

Give me a week or so - I'm catching up from a week of 4.2.1 :).

-- SvenDowideit - 05 Aug 2008

I recon we can - I've not found time, so why let me hold stuff up. If I want something, I'm sure i'm capable of adding it later

-- SvenDowideit - 19 Aug 2008

I've been happy since April....

-- CrawfordCurrie - 19 Aug 2008

Excellent. All are happy. Accepted by concensus.

-- TWiki:Main.KennethLavrsen - 19 Aug 2008

> 2000..2999 plugin javascipt

That pos -- should it really be a number, rather than a symbolic name?

I don't really mind, just think that numbers don't necessarily have long living semantics when everyone piles in on 2000.

A symbolic name such as "plugin-javascript+1" or "plugin-javascript-5" might give better relative positioning.

-- MartinCleaver - 19 Aug 2008

martin, you are reading the old spec. I moved the doc of the plugin impl. up to the top, to avoid further confusions. If you want to see the history, visit old revs of this topic.

-- CrawfordCurrie - 19 Aug 2008

Ok. Thx.

-- MartinCleaver - 20 Aug 2008

ah, ok, I was also put off by that. Now that I read what you actually wanted me to read, I'm not just 'yeah ok, whatever', I'm quite happy

-- SvenDowideit - 20 Aug 2008

There's a related case now, which perhaps should be called ProcessAddToBodyStart - it's needed by Google Analytics the new versions of which now wants to be invoked from the

-- MartinCleaver - 21 Aug 2008

Implemented at Bugs:Item6014. I have added all functions to TWiki.pm, I don't know if this is the right place.

-- ArthurClemens - 23 Sep 2008

I'm not all that well versed in Javascript, however I've noted a few things while working on the ToolTipPlugin. The wz_tooltips.js libraries require that they be included within the body. I tried to add them to the head and the .js library generates an error. The source site - http://www.walterzorn.com/tooltip/tooltip_e.htm recommends that they be added immediately after the body open tag. However the ToolTipPlugin adds them just prior to the closing tag. I didn't change that other than to make the script conditional based upon presence of the tags.

The Yahoo yslow site recommends that for best performance, if at all possible, Javascript should be included at the end of the page. http://developer.yahoo.com/performance/rules.html#js_bottom

So, could any Javascript script management plugin have the alternative of positioning the scripts in the Head, or at the start or end of the body either as required by the library or for performance tuning?

-- GeorgeClark - 23 Sep 2008

That looks fine Arthur. The obvious thing that is missing before a release can be made is..... documentation.

-- CrawfordCurrie - 26 Sep 2008

Great to see some of the many accepted proposals actually being implemented. Wonderful job Arthur.

-- KennethLavrsen - 27 Sep 2008

This has been implemented in Tasks.Item6014. Only needs documentation.

-- ArthurClemens - 28 Nov 2008 - 11:38