Feature Proposal: Enhance TMPL:DEF to allow recursive definition of template elements.

Motivation

We've always wanted to be able to append/prepend elements to TMPL:DEF{head_inner} and TMPL:DEF{action_buttons}

Sometimes plugins/skins want to add something to a TMPL:DEF without having to consider which template/what customisations are in place; and without having to force the user to create a customised skin or hack a default one which will be trashed on every upgrade.

Description and Documentation

Foswiki skin template language allows re-definition of templates. When a template is re-defined using %TMPL:DEF% macro, the previous version of the definition will be renamed to have a ":_PREV" suffix (e.g. redefining template "foo" will cause the previous version to be preserved as "foo:_PREV"). If a template is re-defined multiple times, the older versions will be all preserved with one or multiple ":_PREV" suffixes appended to their name, and the number of occurrences of the suffix represents the generation of the definition. (e.g. if "foo" is redifined 3 times, "foo:_PREV:_PREV:_PREV" will have the orignal definition, "foo:_PREV:_PREV" will have the second generation, "foo:_PREV" will have the 3rd generation and "foo" has the current version).

The preservation of the older version of template definitions makes it possible to inherit content from previous generation when re-defining a template. The inheritance is achieved by simply referring the older version in the new %TMPL:DEF%. Please see the example below.

Original definition in "skin1"

%TMPL:DEF{"mytoolbar"}% format, style %TMPL:END%
%TMPL:DEF{"mywindow"} %TMPL:P{"mytoolbar"}% body footbar %TMPL:END%

In another skin "skin2" that depending on the original skin

%TMPL:DEF{"mytoolbar"}% %TMPL:P{"mytoolbar:_PREV"}%, table %TMPL:END%

This changed the "mytoolbar" template to "format, style, table", while the older version "format, style" is preserved as "mytoolbar:_PREV".

Now in the third skin "skin3" that depending on the previous 2 skins

%TMPL:DEF{"mytoolbar"}% spellchecker, %TMPL:P{"mytoolbar:_PREV"}% %TMPL:END%

Now, "mytoolbar" becomes "spellchecker, format, style, table", "mytoolbar:_PREV" becomes "format, style, table" and "mytoolbar:_PREV:_PREV" is "format, style".

A shorthand macro %TMPL:PREV% is equivalent to %TMPL:P{"<the name being defined>:_PREV"}%, in above case, we can replace the two %TMPL:P{"mytoolbar:_PREV"}% with %TMPL:PREV%.

Please note that, since definitions are processed completely before any expanding is done, redefined macros will always be expanded to their final version no matter if the final version comes before or after the referring of the macro. Take the above case, if we have set skin to "skin3,skin2,skin1", "mywindow" template will be "spellchecker, format, style, table body footbar" even though when "mywindow" is defined, "mytoolbar" is still having its 1st generation.

Other topics talking about this feature:

• ProcessAddToBodyStart
• ProcessAddToHeadAdds
• UpgradeSafeCustomisationsOfSystemTopics

Examples

ChunHuang's syntax

Working patch provided by ChunHuang:

% TMPL:DEF{"foo"}% blah %TMPL:PREV% blah %TMPL:END%

Alternative

% TMPL:DEF{"foo"}% blah % TMPL:P{"foo"}% blah %TMPL:END%

For those less than totally immersed in the world that is Foswiki templates, the former syntax is probably more obvious.

Impact

Whatever the implementation, some performance benchmarking will be required to check for obvious deficiencies.

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: SvenDowideit - 24 Jan 2010, ChunHuang, PaulHarvey

Discussion

I would like to fast track this into foswiki 1.1.

I've pasted the following from Tasks.Item2545.

-- SvenDowideit - 24 Jan 2010

This would be a terrific feature, and it (or a different solution) is badly needed. Sometimes plugins want to add something to a TMPL:DEF without having to consider which template/what customisations are in place; and without having to force the user to create a customised skin or hack a default one which will be trashed on every upgrade.

We've discussed "recursive" TMPL:DEFs before. Rather than specify the PrevVersion with new syntax; if you TMPL:P{"foo"} inside a TMPL:DEF{"foo"}, then Foswiki would "know" that the TMPL:P is talking about inserting the previously defined template def.

Other topics talking about this feature:

• ProcessAddToBodyStart
• ProcessAddToHeadAdds
• UpgradeSafeCustomisationsOfSystemTopics

-- PaulHarvey - 31 Dec 2009

---

I am glad that I am not the only one craving for this feature. Actually, it is very easy to implement. I have made a patch below, in case you are interested, please apply it to lib/Foswiki/Templates.pm and give it try. Instead of using "PrevVersion", a TMPL:PREV will load the previous version.

Index: Templates.pm
===================================================================
--- Templates.pm   (revision 5900)
+++ Templates.pm   (working copy)
@@ -168,6 +168,7 @@
                 $val =~ s/%$p%/$params->{$p}/ge;
             }
         }
+        $val =~ s/%TMPL:PREV%/%TMPL:P{"$template:_PREV"}%/ge;
         $val =~ s/%TMPL:P{(.*?)}%/$this->expandTemplate($1)/ge;
     }
 
@@ -244,7 +245,19 @@
 
             # handle % TMPL:DEF{key}%
             if ($key) {
-                $this->{VARS}->{$key} = $val;
+
+                # if the key is already defined, rename the existing template to  key:_PREV
+                my $new_value    = $val;
+                my $prev_key     = $key;
+                my $prev_value   = $this->{VARS}->{$prev_key};
+                $this->{VARS}->{$prev_key} = $new_value;
+                while ($prev_value) {
+                    $new_value   = $prev_value;
+                    $prev_key    = "$prev_key:_PREV";
+                    $prev_value   = $this->{VARS}->{$prev_key};
+                    $this->{VARS}->{$prev_key} = $new_value;
+                } 
+
             }
             $key = $1;
 
@@ -255,7 +268,19 @@
         elsif (/^END%[\n\r]*(.*)/s) {
 
             # handle %TMPL:END%
-            $this->{VARS}->{$key} = $val;
+
+            # if the key is already defined, rename the existing template to  key:_PREV
+            my $new_value    = $val;
+            my $prev_key     = $key;
+            my $prev_value   = $this->{VARS}->{$prev_key};
+            $this->{VARS}->{$prev_key} = $new_value;
+            while ($prev_value) {
+                $new_value   = $prev_value;
+                $prev_key    = "$prev_key:_PREV";
+                $prev_value   = $this->{VARS}->{$prev_key};
+                $this->{VARS}->{$prev_key} = $new_value;
+            } 
+
             $key                  = '';
             $val                  = '';

---

Guys, what's next? How do I get some attention here? How do I get the feature implemented officially? I need this feature to port my TWiki application into Foswiki and with some upgrade.

-- ChunHuang - 31 Dec 2009

You need to start a feature proposal In development web, and it must then be accepted. Fastest acceptance timeline is 14 days (assuming there are no disagreements on implementation. Somehow I don't think this will be doable for 1.0.8, better target 1.1.

Whoever implements it in core will also need to provide unit tests and documentation.

-- PaulHarvey - 16 Jan 2010

Chun, the next steps are that we allow others to comment for the next 2 weeks, and then we commit code to trunk. We need to add documentation and unit tests to your patch, but its brilliant that you made that already!

I like what you've done, it means that really advanced TMPL coders can not only use TMPL:PREV but also TMPL:P{someotherkey:_PREV}

-- SvenDowideit - 24 Jan 2010

I actually prefer ChunHuang's

% TMPL:DEF{"foo"}% blah %TMPL:PREV% blah %TMPL:END%

syntax over recusing TMPL:P as it shows that the writer specifically intended to recurse - rather than it being a quiet side effect of a possible cut&paste.

-- SvenDowideit - 28 Jan 2010

Well, as we're the only two providing input atm. And you make a good point. I changed the proposal to reflect this.

-- PaulHarvey - 28 Jan 2010

Putting on my anal hat (now, there's a disgusting image), I'm not seeing sufficient specification here. I need to understand exactly what %TMPL:PREV means. For example, if I have a template in "chameleon" skin as follows:

% TMPL:DEF{"foo"}% chameleon fu %TMPL:PREV% chameleon fu %TMPL:END%

Now I define "sausage" skin and write:

% TMPL:DEF{"fnord"}% fnord %TMPL:PREV% fnord %TMPL:END%
% TMPL:DEF{"foo"}% sausage fu %TMPL:PREV% sausage fu % TMPL:P{"fnord"}% %TMPL:END%
% TMPL:DEF{"fnord"}% dronf %TMPL:PREV% dronf %TMPL:END%

Finally I Set SKIN = sausage,chameleon - what should I expect to happen? (I don't want a description of this specific example; I want to see doc that is adequate for a new user to understand what %TMPL:PREV does in all the illustrated cases)

-- CrawfordCurrie - 28 Jan 2010

"Recursiv geht schief" - translates to "recursion is bound to fail". So %TMPL:PREV vs %TMP:P makes no difference.

Reading about a %TMPL:PREV comes a bit as a surprise here. KISS would be not to add another %TMPL macro.

% TMPL:DEF{"foo"}% %TMPL:P{"foo"}% plus some more %TMPL:END%

is what I'd expect. As far as I read this would be equivalent to

% TMPL:DEF{"foo"}% %TMPL:PREV% plus some more %TMPL:END%

Or am I missing something?

Note, that recursion significantly changes the game: we now have a sort of evaluation order which never was part of the game before. If at all "evaluation order" was innate in the skin path and the way %TMPL:INCUDE was processed.

For example, you'd expect

% TMPL:DEF{"foo"}% hello world %TMPL:END%
% TMPL:DEF{"foo"}% % TMPL:P{"foo"}% plus some more %TMPL:END%

to expand to hello world plus some more whereas

% TMPL:DEF{"foo"}% %TMPL:P{"foo"}% plus some more %TMPL:END%
% TMPL:DEF{"foo"}% hello world %TMPL:END%

should expand to hello world only. I think that's what Crawford tries to showcase as well.

There are two potential cases:

1. the same identifier is defined multiple times within the same tmpl file
2. the same identifier is redefined recursively in separate tmpl files, when one TMPL:INCUDEs the other while traversing the skin path

Afaik, the upfront motivation for this proposal is to support (2). But we have to deal with (1) as well as a sideeffect in a way matching expected behavior. I didn't look at the code whether it handles both cases.

-- MichaelDaum - 28 Jan 2010

Cool. Well. I suspect we will have to spec that the order of expansion of multiple DEFs within a single template is undefined (as it is currently when you TMPL:DEF{"foo"} more than once within the same file, or am I mistaken there?).

The main use is supposed to be convenient access to the previous definition in a skin path sense.

Now, whilst trying to write this up in a formal way, it occurred to me that perhaps this could all be turned into something more (admittedly beyond this proposal). Maybe we could build a recursive iterator construct into the TMPL language, provide a way to specify stop conditions - a way of doing LISTVIEW templates maybe? Instead of the crazy %SPLIT% syntax we could have a more crazy recursive TMPL. I'm not enough of a template wizard to see this through though.. This is better discussed in a separate brainstorming topic perhaps.

-- PaulHarvey - 28 Jan 2010

It's undefined in the documentation, but a quick glance at the code shows it's clearly top to bottom (later DEF's overide earlier) which is pretty much what I would have expected. So maybe the doc needs an update.

Every time I encounter one of these %SPLITs I try to replace it with template defs. But there are no cases AFAIK where that equates to a "list template" - normally, the %SPLIT separates head, body* and tail. I can't think of any case for template iteration (though I can think of a strong case for TMPL:IF (which I was too lazy to add when I did TMPL:P{context)=

-- CrawfordCurrie - 29 Jan 2010

SPLIT and REPEAT are the dirtiest corners of the template system. The other place where we have list iterations is in attachtables.tmpl with dedicated head, footer and row definitions, three sets of them to be precise that are used in different contexts. That's very cumbersome and limited to one specific thing. These are then used within the special local ATTACHTABLE macro in attachagain.tmpl. Not to forget about META{attachments} which also makes use of attachtables.tmpl.

Let's be polite and call this loving crap.

That's why Arthur and I independently felt the need to developed ATTACHMENTS in AttachmentListPlugin and similarly in DBCachePlugin=. In both cases iterations is done via our FF (fantastic four) parameters (header, format, separator, footer). Both implementations come with a lot more flexibility wrt iterations and last but not least these can be reused for wiki apps.

The TMPL-world has always been a lot simpler than what we regularly use in the %MACRO-cosmos. There's a fine - and quite arbitrary - border between both where "macros" like ATTACHTABLE look like a real macro but aren't… They are more of a placeholder for a very simple piece of information added while the core processes a very specific screen. You will find these things all over the place. Most of it isn't documented at all.

Iterations can become very specific and people like to control them quite tightly (e.g. excluding, including, skipping and limiting iterations). See also FilterPlugin's FORMATLIST .

That said we should keep TOPIC simple and easy to use.

-- MichaelDaum - 29 Jan 2010

I'm afraid I must remove my name from the CommittedDeveloper field. I have to prioritise my "official" Foswiki dev time on WYSIWYG concerns. I hope our skin developers can find the time to work on unit tests against our current TMPL behaviour to address CrawfordCurrie's concerns.

That is (paraphrasing): We shouldn't modify TMPL behaviour until we understand, document and have unit tests for the existing behaviour.

-- PaulHarvey - 03 Feb 2010

I suspect that the need for this feature is less urgent, assuming ZonePlugin is merged to core. Recursive TMPLs achieve something for skin developers that could also be done with RENDERZONEs, except ZonePlugin provides a way to specify dependencies. What are our thoughts?

-- PaulHarvey - 18 Feb 2010

We need to evaluate what happens at define time and what happens at expanding time. Whenever a new version of TMPL:DEF is created, the previous one is renamed to "oldname:_PREV". For example, assuming "fnord" is not defined, and "foo" is defined in earlier skin foo. And skin=sausage, chameleon

% TMPL:DEF{"fnord"}% fnord %TMPL:PREV% fnord %TMPL:END%
% TMPL:DEF{"foo"}% sausage fu %TMPL:PREV% sausage fu % TMPL:P{"fnord"}% %TMPL:END%
% TMPL:DEF{"fnord"}% dronf %TMPL:PREV% dronf %TMPL:END%

Originally

fnord => (empty, undefined)
foo => chameleon fu chameleon fu

After the first 2 statements

fnord:_PREV => (empty, undefined)
fnord => fnord % TMPL:P{"fnord:_PREV"} fnord
foo:_PREV => chameleon fu chameleon fu
foo => sausage fu % TMPL:P{"foo:_PREV"} sausage fu % TMPL:P{"fnord"}%

After the 3rd statement

fnord:_PREV:_PREV => (empty, undefined)
fnord:_PREV => fnord % TMPL:P{"fnord:_PREV:_PREV"}% fnord
fnord => dronf % TMPL:P{"fnord:_PREV"}% dronf
foo:_PREV => chameleon fu chameleon fu
foo => sausage fu % TMPL:P{"foo:_PREV"} sausage fu % TMPL:P{"fnord"}%

If you understand this, you should be able to deduce what will be the value when they are referenced.

-- ChunHuang - 02 Mar 2010

OK, that's excellent. There's sufficient there to remove my concern, assuming it gets into documentation (together with the assertion that the evaluation is top-down in the order that template files are read). I like the idea of treating the defs as a stack, though I might have expected some way to pop that stack (restore %TMPL:P{"fnord:_PREV"}% as %TMPL:P{"fnord"}%, and %TMPL:P{"fnord:_PREV:_PREV"}% as %TMPL:P{"fnord:_PREV"}% etc)

-- CrawfordCurrie - 02 Mar 2010

Well, it wasn't that complicated in my vision. I didn't think of popping the stack, probably because I didn't feel the need at the time. For implementing that, I think we will need to introduce a new tag %TMPL:POP% and a few lines of code in Templates.pm. But, I am still not very clear on why we need such "pop".

I am very glad that the topic was evolving even when I didn't pay attention to it. It proves that the community is really active.

-- ChunHuang - 02 Mar 2010

Chun, TMPL:POP and fancy features was not part of the original proposal and would need people to have time to consider this.

If we really do not need this and the TMPL:PREV suits the purpose without preventing further development of a POP later then I suggest this

• We declare the TMPL:PREV feature accepted by consensus.
• Chun writes an updated spec that carefully exlains the new feature. This spec is needed to document the new feature so users can use it.
• Chun checks in the updated code WITH the updated documentation. Use Tasks.Item2545
• Any POP ideas are deferred to the future and would be proposed with spec in a new feature proposal when someone gets the time and desire for it.

-- KennethLavrsen - 02 Mar 2010

I assumed that updating the spec basically means updating the "Description and Documentation" section. So here you go, a new "Description and Doumentation". Please let me know if am being silly.

-- ChunHuang - 02 Mar 2010

That's fine. I wasn't seriously asking for stacks; just observing that I like the idea that we could have that feature some time in the future, should we ever need it. I don't see any short-term application, though.

This is an excellent feature, I look forward to using it!

-- CrawfordCurrie - 03 Mar 2010

I've commited your code as is, plus a simple unit test and barely minimal documentation. More docco should be added..

congrats

-- SvenDowideit - 26 Mar 2010

What if the same template file is read twice? Before Foswiki was robust against this. However with recursive tempalte refinements, this could end up in a mess.

-- MichaelDaum - 15 Apr 2010

um, easy, raise a bug (a new task), write a unit test that shows how the processing has changed from 1.0 to 1.1 and then see what we all think of the issue you claim to have found.

could isn't particularly useful to us - unit tests are.

-- SvenDowideit - 15 Apr 2010

I am just asking to make sure and to understand. Is there a kind of protection not to reload the same tmpl file twice? I sensed that this becomes crutial when tmpl definitions are recursive.

I was tracking down Tasks.Item8759 of the current AutoViewTemplatePlugin where it uses readTemplate() to find out if a template actually exists before applying it.

-- MichaelDaum - 15 Apr 2010

Probably required when debugging. Probably overkill in production (I would hope).

-- CrawfordCurrie - 15 Apr 2010

As users can write view temlates online, this recursion protection is a must as it most probably is a DoS vector.

-- MichaelDaum - 15 Apr 2010

Good point. Flexibility always has a price

-- CrawfordCurrie - 16 Apr 2010

What is the status of this feature? I can't find any documentation on this.

Tasks.Item10807 implements a recursion check, current limit at 999 levels. Normal web search seems to need 15 levels.

I would not ever user _PREV myself: it too easlity leads to infinite recursion. %PREV% is much safer.

-- ArthurClemens - 28 May 2011