Feature Proposal: here-document syntax for macros

Motivation

Need to break the parameter blockage

Description

Support here-document syntax for macro parameter values.

For a long time we've had a problem with macros such as

%IF{"condition" then="something really long" else="something with newlines"}%

Several solutions to this problem have been proposed, and some even implemented (GluePlugin) but all are, for one reason or another, inflexible or visually horrible, or not integrated (or too hard to integrate) in the right places (such as the meta-data parser), or eat newlines.

The proposal here is to take the shell/perl "here document" syntax into TML, to support such long parameter values.

here-document syntax has the advantages that it is:

1. familiar to anyone who has used the sh shell family, perl, or any one of a number of other scripting languages
2. easy to parse and extract
3. consistent with "line-oriented" TML
4. easy on the eye
5. provides a way to unambiguously "tag" sections in the raw topic text

A consistent extension of macro parameters to support this syntax would instantly support all plugins that use registered tag handlers.

Note that MichaelDaum solved this problem with the GluePlugin, and his syntax should be considered as a realistic alternative to this proposal. The main reason not to go the GluePlugin route is that the semantics of the GluePlugin are already established and in use in many applications. I'm talking about doing something right at the very lowest level of macro parsing, way below where the GluePlugin sits, and re-using the same syntax would change the rules for those apps. However this proposal would not prevent use of that plugin for those who want it.

Update: This proposal originally specified that macros embedded in here-documents would be evaluated before the macro call, just like regular quoted macro parameters (see rev 36). The proposal now calls for macros embedded in here-documents not to be evaluated before the call i.e. here-documents provide a way of delaying macro expansion. Thus, here-documents provide a syntax for changing the macro evaluation order.

Documentation

Extended macro parameter values

Macro parameters can be specified either as quoted values (using double quotes) or as here-documents. A here-document is a block of text, terminated with a special string, that is associated with a macro parameter. For example,

%SEARCH{"what" excludetopics=<<EXCLUDE}%
Web*,
*Form
EXCLUDE

You can have more than one here-document on a single line. They are read in the order they appear in the macro parameters:

%IF{"condition" then=<<ELSE else=<<FI}% %SEARCH{<<SEARCH}%
then-clause
ELSE
else-clause
FI
search-for-this
SEARCH

In this case, the value of the then parameter is then-clause, the value of the else parameter is else-clause, and the value of the search string is search-for-this. Note that:

1. The closing string may be any sequence of one or more alphanumerics (A..Z, a..z, and 0..9)
2. The closing string must start at the beginning of a line (in the raw topic view).
3. The newline at the end of any line containing a here-document (<<HERE) will be ignored. The value starts after this newline. Example

    If it takes %IF{"condition" then=<<ELSE else=<<FI}% men to dig a
    three
    ELSE
    four
    FI
    hole.
    

   will expand to "If it takes three men to dig a hole"
4. The newlines before and after the closing string will be ignored.
5. If you miss out the closing string, the value will be the whole of the rest of the topic. This is an error condition, and so no macros will be expanded in that value (all % will be escaped) and an inline alert will be emitted to draw attention to the problem.
6. You could use the same closing string for two or more here-documents on the same line, but don't do that; it would be confusing.

Embedded macros inside here-documents are not expanded according to the same rules as apply to normal macro parameters. Embedded macros inside here-documents will only be evaluated after the containing macro has been evaluated. You can have a here document inside another here document, though the closing strings obviously should be unique (it would work if they were the same, but that would be confusing). Be warned that if expansion of an embedded macro generates the terminator for the here-document, it will not close it; the terminator has to be explicit (and has to be in the same topic as the macro call). For example ( complex example for clarification!),

    * Set X = FORMAT
    * Set Y = \"every\"
%SEARCH{<<STRING format=<<FORMAT}%
%Y%
STRING
%X%
$topic
%SPACEOUT{<<inner}%
$topic
inner
FORMAT

is equivalent to %SEARCH{"$percntY$percnt" format="$percntX$percnt$n$topic$n$percntSPACEOUT{$topic}$percnt"}%.

Note that in the above example, %Y, %X and %SPACEOUT will all be evaluated after %SEARCH is called. If you want macros in values to be evaluated before the call, use regular quoted macro parameters.

Note that the first and last newlines are stripped off the block of text. So if you want to specify a parameter value that starts and ends with a newline, you have to give a blank line. For example,

%IF{"condition" then=<<ELSE else=<<FI}%

then-clause

ELSE
else-clause
FI

The default parameter

The "default", or unnamed, parameter is also supported, thus:

%AMACRO{param=<<ONE}% %BMACRO{<<TWO param=<<THREE}%

Here-documents don't eliminate $dollar completely

The classic example of a complex parameter value is the nested search example, repeated here for reference:

%SEARCH{ "culture" format="   * $topic is referenced by:$n      * $percntSEARCH{ \"$topic\" format=\"$dollartopic\" nosearch=\"on\" nototal=\"on\" separator=\", \" }$nop%" nosearch="on" nototal="on" }%

Here-documents make it possible to eliminate most of the escaping, it is still necessary to escape the $topic in the inner SEARCH's format:

%SEARCH{ "culture" format=<<HERE nosearch="on" nototal="on" }% 
   * $topic is referenced by: 
      * %SEARCH{ "$topic" format="$dollartopic" nosearch="on" nototal="on" separator=", " }%
HERE

Impact

WhatDoesItAffect:

%WHATDOESITAFFECT%

edit

Implementation

-- Contributors: CrawfordCurrie, MichaelTempest, ArthurClemens, SvenDowideit, PaulHarvey, LarsEik, IsaacLin

Discussion

Can I use a macro to mark the end-point?

Uncomment one of these:
   * Set MARKER=LITTLE
   * #Set MARKET=LOTS
%MACRO{firstpart=<<%MARKER% secondpart=<<ALL}%
blah
LITTLE
bzzzt
LOST
blat
ALL

No

Can I use the same end marker?

%AMACRO{<<END}% %BMACRO{<<END param=<<END}%
one
END
two
END
three
END

No

Can I nest HERE documents?

%AMACRO{<<ENDA}%
apple
%BMACRO{<<ENDB}%
parameter for BMACRO
ENDA
more content in the BMACRO parameter
ENDB
orange
ENDA

Yes - as long as you use unique end markers

• Clarification - I intended that the first ENDA (and the line after it) be included in the parameter to BMACRO. -- MichaelTempest - 13 Oct 2009

-- MichaelTempest - 12 Oct 2009

man, thats pretty horrible. will we also then need the 'END' and "END" variation to cope with different quote interpolation?

or is there something better. like for example the already existent named SECTIONS.

my alternate proposal is :

If it takes %IF{"condition" then="$include(section=then)" else="$include(%WEB%.%TOPIC% section=then)"}% men to dig a
%STARTSECTION{then}%three%STOPSECTION{then}%
%STARTSECTION{else}%four%STOPSECTION{else}%

where we leverage the idea that $include(web.topic section=name) is processed after the IF macro is evaluated.

this 'format' specifier is actually on my list of things to implement for SEARCH etc, but really it applies here too.

-- SvenDowideit - 12 Oct 2009

But we do need something that is usable by end users. You know, end users can make applications themselves? So far I think the 'here' syntax is easier to understand than the sections.

-- ArthurClemens - 12 Oct 2009

mmm, I'm somewhat surprised by your reaction Arthur.

I've found that in languages that it exists in the <<HERE syntax is considered pretty difficult for users - and seems not to be liked by the early/intermediate very much. I'm not sure that its easier to understand - especially when you consider the weird restrictions that generally come with it - like that the STOP word needs to be at the beginning of a line.

seems to me that the issue might be that that sectional syntax is uglier than it should be (given its rather horrid design my committee history)

-- SvenDowideit - 12 Oct 2009

I am not totally happy with 'here', but it is better than current alternatives, especially the verbose INCLUDE / SECTION syntax. Regarding the requirement of starting on a new line, that also goes for TML headers, bullets and tables.

Not to say we need to stop thinking right away, there might be a better syntax even.

-- ArthurClemens - 12 Oct 2009

Sven has a very valid point. I'm not great fan of here-document syntax either. However, aside from the attraction is that it's easy to retro-fit to existing macros, there's a deeper reason to pursue this approach. The SECTION syntax has a major problem, because it uses strings that are recognisable as macros and should therefore be expected to be subject to the same expansion rules as other macros. This makes static (context-free) parsing of topics impossible. There are a number of hacks in the core that make it possible to support %SECTION blocks, but they are ugly and inefficient. If we had thought of this a long time ago, we might well have implemented sections like this:

%SECTION{<<AUTH name="authmessage"}%
You are not authorised
AUTH

My gut feeling is that this is the right place to do this; the syntax I agree is less than ideal, and I'm happy to listen to alternatives.

Regarding quotes; I don't think anything special is required. Logically I'm talking about replacing "this" with <<THISthisTHIS, but otherwise treating this identically with the one exception that it becomes possible to include double quotes i.e. I was assuming that the here document string would be treated identically to any other macro parameter, in that it would interpolate any embedded macro.

-- CrawfordCurrie - 13 Oct 2009

Where I use GluePlugin most is in nested format strings or strings that result in usage of $percnt escaping.

The example for nested HERE syntax, in my opinion, is not very intuitive.

-- PaulHarvey - 13 Oct 2009

The nested HERE example was intended to explore corner cases to see if/how it affects the specification, because users (including myself) will try the oddest of things.

I have a minor comment on the use of SECTIONs instead of HERE documents. SECTIONs are included in the topic output, so this:

If it takes %IF{"condition" then="$include(section=then)" else="$include(%WEB%.%TOPIC% section=then)"}% men to dig a
%STARTSECTION{then}%three%STOPSECTION{then}%
%STARTSECTION{else}%four%STOPSECTION{else}%
hole

would produce something like this:
If it takes three men to dig a three four hole

This is probably not what was intended, and I am sure something could be done about it. I don't have any suggestions on that.

-- MichaelTempest - 13 Oct 2009

MichaelTempest, your nested HERE example opened ENDA once but ended it twice. I'm unsure what the result is supposed to look like.

• I added a clarification -- MichaelTempest - 13 Oct 2009

I haven't had the need to do this in other environments before, but I would hope that nested HERE documents could work as follows:

%SEARCH{ ... format=<<SEARCH footer="..."}%
Blah *foo* $percntCALC{<<CALC}$percnt
$IF($EXACT($formfield(calc), awesome), flying pigs, snafu))
CALC
 and some more _careful_ formatted search stuff.
SEARCH

Yes

-- PaulHarvey - 13 Oct 2009

On reviewing MichaelTempest's nesting example, I'm not so sure any more. I suspect it's no more technically complex to accept recursive definitions of here documents than it is to disallow them; so it comes down to what people think is best. Opinions? Should <<ENDA close on the first ENDA, or the second?

-- CrawfordCurrie - 13 Oct 2009

Because it would be best for clarity to not reuse a terminator that needs to be included somewhere within the text, and it should always be possible to find a suitable terminator, I suggest ending the included text on the first ENDA. It's easier to document (no special nesting rules required) and encourages the use of different terminators.

-- IsaacLin - 13 Oct 2009

As a user I would suggest:

$here(DING) some text and new lines $here(DING)

This would be better than traditional HERE, if the core could swap $here with >>HERE and match the ending HERE? We are used to the $ and % so it would make sense for me at least. If it is possible or sensible programmatically I can't say.

-- LarsEik - 13 Oct 2009

Something analogous:

%SEARCH{"what" excludetopics="<EXCLUDE>"}%
EXCLUDE
Web*,
*Form
EXCLUDE

and

If it takes %IF{"condition" then="<ELSE>" else="<FI>"}% men to dig a
ELSE
three
ELSE
FI
four
FI
hole.

or even

If it takes %IF{"condition" then="<ELSE>" else="<FI>"}% to dig a
ELSE three "youngsters" ELSE FI four "boomers" FI hole.

-- ArthurClemens - 13 Oct 2009

I'm not sure the example immediately above is very intuitive either. Also, it's almost functionally exactly the same as a pair of START/ENDSECTION tags.

Rather than got to a brand new syntax to achieve that, I'd prefer we enhanced START/ENDSECTION and INCLUDE.

If standard SECTIONs could be hidden from normal page view (only rendered when explicitly INCLUDEd) that would go a long way to helping the string soup situation.

Of course, you still can't use something like:

If it takes %IF{"condition" then="%INCLUDE{section="ELSE"}%" else="%INCLUDE{section="FI"}%"}% to dig a
%STARTSECTION{"ELSE"}% three "youngsters"
and
some

newlines
%ENDSECTION{"ELSE"}%
%STARTSECTION{"FI"}% four "boomers" %ENDSECTION{"FI"}% hole.

Because the newlines can break the then="" attribute.

Maybe we can use a hybrid:

If it takes %IF{"condition" then=<<ELSE else=<<FI}% to dig a
%STARTSECTION{"ELSE"}% three "youngsters"
and
some

newlines
%ENDSECTION{"ELSE"}%
%STARTSECTION{"FI"}% four "boomers" %ENDSECTION{"FI"}% hole.

Then what about sections from other topics?

If it takes %IF{"condition" then=<<ELSE "WebName/SubWeb.SomeTopic">> else=<<FI>>}% to dig a

My hybrid example there needs some more thought (do we really need topic path in double quotes? What about international character sets? does it make sense to require that << be closed with >>?).

I think new syntax for passing in named sections as attribute parameters is warranted, because it is logically distinct from the string soup we've had to worry so much about in the past.

But I think it would be a shame if we had two different syntaxes for declaring sections.

Just my opinion

-- PaulHarvey - 14 Oct 2009

Re-using the SECTION syntax is an interesting option. It doesn't quite achieve what I had in mind, because it isn't context free. Because the <<HERE syntax is liberated from macro processing loop, it lets us write things that are quite simple and relatively readable. For example,

%WHILE{"condition1" do=<<ENDWHILE}%
do-this-1
<<ENDWHILE
%WHILE{"condition2" do=<<ENDWHILE}%
do-this-2
<<ENDWHILE

and the ENDWHILE's are unambiguously associated with the preceding <<ENDWHILE. However using SECTION syntax I can do this:

    * Set confuseme = endwhile
   * Set DO = STARTSECTION
   * Set OD = ENDSECTION
%WHILE{"condition1" do=<<%confuseme%1>>}%
%%DO%{"%confuseme%2"}%
do-this-2
%%OD%{"%confuseme%2"}%
%WHILE{"condition1" do=<<%confuseme%2>>}%
%%DO%{"%confuseme%1"}%
do-this-1
%%OD%{"%confuseme%1"}%

This is because %SECTION is a macro. All macros are (theoretically) asynchronously processed everywhere in the topic at once (yes, I know that's not how it works, but it's how it should seem to work), and you can express new macros in terms of other macros. That means you have to read the whole topic before you can process a single macro. You can see that as an advantage, because it gives you tremendous expressive power. However it is fairly inefficient in the core code, and can be made horrendously inefficient (and amazingly unreadable, as in the example above) by sloppy use of the macros by the end user.

The here-document syntax, on the other hand, is processed in a linear order in the topic, top to bottom, without any information about the execution context or expansion values of other macros. That makes it extremely efficient and unambiguous, though clearly it is nowhere near as flexible as using macros.

BTW another point not expanded on previously is that here-documents can be used to "tag" sections in a document. If I can write:

%MACRO{<<SEC2}%
blah blah
SEC2

then an external processor (e.g. a section editing plugin) can come along later and pick out SEC2 as a target for editing. Because it doesn't depend on macro processing, the here-document is easy to identify and extract from raw topic text.

-- CrawfordCurrie - 14 Oct 2009

Well, those are very good reasons not to use SECTIONs.

I'm glad you bring up sectional editing, because it's something on my mind. I would like to ask that we take this opportunity to carefully consider sectional editing - or the more general problem of being able to address/target sections of topics without having to process every macro in it, because it would be a shame if we had STARTSECTION, HERE, and then later some other syntax to mark up topic sections.

What are our thoughts on extending the HERE notation to enable addressable topic sections? I suspect it would be desirable to have start/end delimiters in this case, but perhaps it's acceptable to assume an implied start delimiter after the appearance of a <<SEC.

I still have some reservations about the syntax being too alien from %foo% style macros. But I suppose %%SECNAME%% or %<SECNAME>% is too ugly...

-- PaulHarvey - 14 Oct 2009

HERE documents are a nice idea, I like it, as long as it is kept simple. As with all of the features of Foswiki, users have to make the right choice from a conglomerate of possibilities. Most of the time they fail to steer into the right direction. All too often I come to see a bunch of wiki apps that aren't maintainable anymore and people need a full-fledged workshop to open their eyes. That's because (a) documentation sux and (b) there are too many roads to rome, some go knee-deep through the mud.

This proposal tries to address one of the weaknesses of TML. Lots of wiki apps got their main logic inside params to some %MACRO. So moving it out - for instance using HERE or INCLUDE is a good practice. That way multiple levels of escapes like $dollardollarpecnt can be omitted. However, from what I was able to get reading the stuff above, we still have these standard escapes in the HERE documents even though they are outside of the normal %MACRO{param} nesting. I wonder if it is possible to diverge from the normal left-to-right-inside-out in a way to get around this problem.

-- MichaelDaum - 14 Oct 2009

(Michael's question is addressed in UseSyntaxToChangeEvaluationOrder)

... and finally I can add a deprecation flag to IfDefinedPlugin with this elegant way of conditional sections (together with the new OP_match for queries).

-- MichaelDaum - 14 Oct 2009

How would line-continuation characters be handled? Which END would mark the end of then, first or second?

%IF{"condition" then=<<END}%
This is not the \
END

This is
END

-- MichaelTempest - 14 Oct 2009

I did some experiments on the "HERE-document" implementation, and it's a lot harder than I'd thought to do it efficiently. Not given up hope yet, though. I moved the expansion order discussion to a different feature proposal (UseSyntaxToChangeEvaluationOrder)

Micael, the thing about HERE-documents is that they take everything as verbatim up until the closing string. So line continuation chars would be sucked up with everything else.

-- CrawfordCurrie - 05 Nov 2009

I added what I feel could be related topics. It would be nice to avoid too many different ways of addressing or otherwise marking up topic content. I feel that these different goals could align into a common solution by way of enhanced TOM spec.

Updated TopicObjectModel

-- PaulHarvey - 06 Nov 2009

I did an experimental "HERE-document" implementation, and I found that it was easiest to give delayed expansion to the HERE documents. I also think this is the most useful form of HERE-document, so I focussed on that first. I encountered something I had not thought of, so I thought I should mention it here.

I looked at the classic nested search example. I found I could eliminate most of the escaping, but I had to escape $topic in the inner SEARCH's format, if I wanted to use only standard features/plugins:

%SEARCH{ "culture" web="System" format=<<HERE nosearch="on" nototal="on" }% 
   * $topic is referenced by: 
      * %SEARCH{ "$topic" web="System" format="$dollartopic" nosearch="on" nototal="on" separator=", " }%
HERE

I could get rid of all escaping with the help of EasyMacroPlugin:

%REGISTERMACRO{"INNERSEARCH" format=<<END_INNER_FORMAT param="uses" uses="%TOPIC%"}%
%SEARCH{ "$uses" web="System" format="$topic" nosearch="on" nototal="on" separator=", " }%
END_INNER_FORMAT
%SEARCH{ "culture" web="System" format=<<END_OUTER_FORMAT nosearch="on" nototal="on" }% 
   * $topic is referenced by: 
      * %INNERSEARCH{uses="$topic"}%
END_OUTER_FORMAT

Another consideration is TMCE/WysiwygPlugin. The end-of-here-document marker must be kept on a line on its own after wysiwyg edit.

-- MichaelTempest - 23 Jan 2010

Here is the diff for my implementation of here-documents:

• HereDocuments.diff

-- MichaelTempest - 24 Jan 2010

I have changed my implementation since, but the diff is still in-principle correct.

Nested macros using HERE-documents where the end-marker is missing or in the wrong place have the potential producing consuming vast amounts of memory. This is quite easy to do by accident. For example, this made my server consume several hundred megabytes (the first <<FORMAT should be <<FORMAT2 )::

%FOREACH{<<WORDS2  format=<<FORMAT}%
1,2,3,4,5,6,7,8
WORDS2
%FOREACH{<<WORDS  format=<<FORMAT}%
ThisIs,OneWiki,WordPer,ParaMeter,OfCourse,PlentyNew,ExamplesCould,BeFound
WORDS
   * %SPACEOUT{$topic$dollartopic}%
FORMAT
FORMAT2

To prevent that, I changed the code to detect a missing HERE-document marker, emit an inline alert and convert all % in the HERE-document to %.

I updated the proposal to delay expansion of macros in here-documents and to provide for this way of handling missing end markers.

I have added my name as committed developer.

-- MichaelTempest - 24 Jan 2010

Suggest we use double or single quoted strings as inspired by other programming languages.

%FOO{<<"HERE"}%

for normal inside-out

%FOO{<<'HERE'}%

for delayed

-- PaulHarvey - 25 Jan 2010

Unfortunately, Foswiki already supports the use of single-quotes for regular macro parameters, and single-quotes mean the same as double-quotes (i.e. %FOO{param='%BAR{$topic}%'}% means the same as %FOO{param="%BAR{$topic}%"}% ) and this is already supported on the release branch (dunno how long it's been there, though).

So using single quotes for delayed evaluation only for here-documents would be inconsistent, and using it for delayed evaluation for all parameters could break existing wikis.

-- MichaelTempest - 25 Jan 2010

Excellent work, thanks Michael. Proper escapes and single quotes were added a long, long time ago, when I tried to rationalise the way strings and escapes were supported. I was beaten up (and down) so it never went properly "live", though it persisted in the code as "context free" or "friendly" syntax. The friendly syntax is only used by %IF in the core, where I used it with the intention of enabling late evaluation of the then and else clauses. It is also used in a (small) number of plugins, including the Action Tracker; this is the main reason I have never removed the support again.

I am not in favour of using quotes the way Paul describes. I always found it horribly nerdy in perl. IMHO everything between the <<HERE and the closing HERE should always be taken as verbatim, and the decision whether to early-or-late eval that string only taken where the here-doc has been 'converted' to a standard param and 'fed' to the '=' operator. If you think that way, you can see it's easy to replace = with another operator for late eval, as discussed in UseSyntaxToChangeEvaluationOrder.

-- CrawfordCurrie - 25 Jan 2010

A note to Michael T as follow up to a remark on IRC: This feature should never hit Release01x00 branch. The 1.0.X releases should from now on only be urgent bug fixes. We should never put large enhancements or syntax/TML changes into a patch release. In fact I do not plan to release a 1.0.10 unless a securiy or very urgent fix comes along.

We should focus on getting a 1.1 out. We will soon create a Release01x01 branch when the last features are coded and the last step of getting stability starts. And then we will be able to test this feature in several betas and RCs.

You can check code into trunk before the 14 day period is over as long as you are prepared to revert all the changes if the proposal is rejected.

-- KennethLavrsen - 25 Jan 2010

It's hardly just perl that uses double quotes for interpolated strings and single for non-. But I can appreciate we shouldn't limit our imaginations to the mind of programmers, hopefully our wiki syntax is usable by non-programmers too.

As a first suggestion of an assignment operator, how about |= ?

I am still concerned we are unnecessarily creating a new syntax for defining sections. Eventually it would be nice to DOM-ify Foswiki's markup. Index, query, mash-up SECTIONs. SvenDowideit's idea was to keep existing param syntax but introduce an $include(section) token which I assume would be parsed/expanded by foswiki before handing the param on to a plugin.

-- PaulHarvey - 26 Jan 2010

There are problems with the current (experimental) Here-document support

I am still giving thought to Sven's $include() idea. Suppose we went the $include() route - what should this do?

%SEARCH{ ... format="$include($topic)" }%

This example is not meant to be specific to %SEARCH. From a user's perspective, the format above should mean the same as format="$percntINCLUDE{$topic}$percnt". However, $include is supposed to be part of a generic mechanism, and $topic is macro-specific. For it to work correctly, $topic should be expanded before $include, but $include is supposed to be expanded before the macro handler is invoked.

I am not trying to shoot down $include() - but I do not know see how to work around this issue. Suggestions?

-- MichaelTempest - 04 Feb 2010

The idea with $include is that it is evaluated during (and at the end) of the formatting. So as you say, format="$include($topic)" is equivalent to format="$percntINCLUDE{$topic}$percnt".

whereas in the case where you want the expansion to happen first, you are already able to use the non-escaped %!INCLUDE{} - basically, INCLUDE{$topic} does not make sense, as you can't ask to evaluate an INCLUDE before you know what you're including.

to sum up - not, $include is not supposed to be expanded before the Macro handler is invoked - thats what INCLUDE already does.

Sadly, I still can't work out what functionality this HERE syntax adds - it seems to me to just add a shorthand version of what we already have.

I would add that it would be nice to have a 'default' for INCLUDE such that if it does not specify a topic, but does have a section, that BASEWEB.BASETOPIC is used.... and that STARTSECTION have a hidefromrendering option.. (as proposed by Arthur)

-- SvenDowideit - 15 Feb 2010

The advantages are described at the top of this topic. it would, for example, have saved my life on a (tm)wiki app I was just working on for a client, where I needed to be able to pass newlines in a parameter that didn't support $n (or any other formatting macros).

-- CrawfordCurrie - 16 Feb 2010

HERE documents really would be very handy. It would dramatically reduce the amount of yada. Example:

%ADDTOZONE{"body" web="%WEB%.%TOPIC%" section="initjs"}%
<verbatim style="display:none">
%STARTSECTION{"initjs"}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world");
});
</script>
%ENDSECTION{"initjs"}%
</verbatim>

would become

%ADDTOZONE{"body" text=<<HERE}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world");
});
</script>
HERE

This is a lot simpler. It is not that HERE invents a construction that is impossible to put together using named sections. It is that it gets a lot more handy.

-- MichaelDaum - 16 Feb 2010

My position on this now is not the same as when I started.

(1) I think the "newline" argument might be void, as I discovered whilst writing the ExpandMacroTests that Foswiki handles newlines in regular parameters too. For example, Michael's example could be implemented as

%ADDTOZONE{"body" text="<script type=\"text/javascript\">
jQuery(function() {
  alert(\"hello world\");
});
</script>"}%

- but here-documents give you newlines in parameters with minimal cruft

(2) Here-documents not only have less visual clutter than named sections, but they have no name. This is rather like blocks in other programming languages. I don't want to have to name every block - whether I program in C, pascal, perl or TML.

(3) Here-documents are syntactic sugar.

(4) Here-documents are conceptually simple - just another form of quoting, which may already be familiar to newcomers to Foswiki. This should be helpful to people new to Foswiki because it gives a low hurdle to escape-free quoting. If I am struggling with a complex formatted search or something with %IF or %FOREACH, then I may not have enough free brain space to go and learn how to use %SECTION and %INCLUDE to be able to solve my quoting problems, to be able to build the complex TML, to be able to solve my real application-domain problem.

-- MichaelTempest - 16 Feb 2010

@Crawford - if you only needed newline stuff, you have always been able to do that using named sections

@MichaelD - yes, i realise that, but there is an implication that HERE doc adds new functionality - and so far, i think this is untrue. If we're after simplifying the typing, then I would structure the proposal differently, and not import another parsing system into the rendering process. (additionally, the proposal docco on this topic goes a long way to confuse the issue, if all it is, is a typing shortcut).

@MichaelT - that still sounds like an argument to simplify the (really crappy) SECTION and INCLUDE syntax to make it more user friendly, both for new users and for those of us that do it alot.

So, we've finally reached the point where I've read enough to raise a concern - I am now pretty sure that this proposal only raises an issue of typing and cruft, and that creating a completely new syntax that reserves \nSOMETEXT for the parser at some random time, and further entrenches the un-DOMability of the TML source.

-- SvenDowideit - 16 Feb 2010

I don't think anyone ever claimed that this proposal added new functionality; perhaps you are confusing it with the discussion on late expansion of macros in parameters. As I read it, none of us is particularly overjoyed by the heredoc proposal. Such a syntax is not beautiful, and if we were designing TML from scratch, this isn't what we'd do. However it is pragmatic. Michael has done some great work creating an implementation that lets you analyse the behaviour in detail. Perhaps the proposal is too verbose; it has, after all, been in the process of distillation for some considerable time. As far as I can see, no viable alternatives have been proposed (most people don't consider %SECTIONal include as a viable alternative).

"further entrenches the un-DOMability of the TML source" is IMHO pure FUD. Heredocs have been implemented into the macro parser, which IMHO means that they are actually more DOMability-friendly than any so-far-proposed alternative (including SECTIONs, which are glued on to the syntax in a horrible way). Sven, it's fine to raise concerns, but when you do so, please give specific counter-examples showing how the problems the proposal addresses can be solved simply and cleanly. If that is indeed the case, then your concern is valid and your examples can be used to document the alternative. Right now you are obviously seeing something that the rest of us are missing, but you are failing to communicate it in a way that can be understood and addressed.

-- CrawfordCurrie - 17 Feb 2010

Let me start by saying sorry. I thought we were using the Feature process to try to get the best possible result, and specifically that adding new language idioms was something that we should do very conservatively. (As opposed to idioms that are already used).

Crawford, you seem to be arguing that its my fault for not working out your proposal faster - and that somehow its my responsibility to tell you what I know - whereas I've spent the time since you proposed it, trying to work out why your proposal is so special - rather than a very simple syntactic sugar it (4 months later) appears to be. During that time I've tried to ask a few times, but got responses like The advantages are described at the top of this topic - which can't really be any less helpful.

Now on to what I see on this proposal.

First up, the motivation says Need to break the parameter blockage - nothing there about adding a shorthand for an existing functionality.

The Description goes on to tell us that solutions to this problem have been proposed, and some even implemented (GluePlugin) but all are.... again, nothing talking about a shorthand.

If the proposal were specifically to be a syntax shorthand for INCLUDE, we would see a very simple proposal, showing the longhand way to write it, and then the shorthand - no confusion about what will happen - however, from what I've worked on below, it would result in a completion of the INCLUDE & SECTION macros..

further objections

general malaise

I was expecting that many people seeming unhappy with the HERE syntax specifically should be enough to block the proposal already - at least until more ideas have been brought up.

• I am not totally happy with 'here', but it is better than current alternatives - Arthur
• I'm not great fan of here-document syntax either. - Crawford
• The example for nested HERE syntax, in my opinion, is not very intuitive. - Paul

unDOMability of the topic source

un-DOMability - I haven't seen how the HERE sections will be protected in WysiwygPlugin, and am pretty curious how that will affect attempts to make macro editors such as ComponentEdit work - I guess that'll be something that will be addressed in this proposal topic? I guess I was unclear - I'm not talking about the DOM ability of the resultant output, I'm talking about the DOMability of the source, and the attempts that we used to make to become more bidirectional.

lack of reusability of sections

Sectional includes encourage reuse - they already remove the need for multi-level escaping (via include params) and provide an avenue into allowing multi-lines. I don't think the existing proposal will improve on this.

HERE syntax is a pretty limited scope thing, which i think that we're better of treating it as a simplification of the broader transclusion mechanism, rather than creating a new 'short range' transclusion

not sure this is good or bad

it would, for example, have saved my life on a (tm)wiki app I was just working on for a client, where I needed to be able to pass newlines in a parameter that didn't support $n (or any other formatting macros)

this is basically saying that the pervasive-ness of this feature is better than us moving foswiki forward so that all parameters support newlines, $n, utf8, who knows what else

makes me wonder - maybe we should actually replace all parameters with a simplification of the here doc..

so replace

%MACRO{
   "one" 
   text="two" 
   param="three" 
   another="four"
}%

with

%MACRO{ALLHERES}%
one
_DEFAULT
two
TEXT
three
PARAM
four
ANOTHER

rather than fixing up the parsing of TML

I wonder if this is pretty much like non-type checked, un-named parameters

if we imagine a scenario where we have many multiple of HERE's (is there a term that you can actually talk about?)

%ADDTOZONE{<<WHERE text=<<TEXT param=<<PARAM another=<<ANOTHER}%
one
WHERE
2
TEXT
true
PARAM
on
ANOTHER

and then a new person wants to change the output, they refactor to make it more readable

%ADDTOZONE{
    <<WHERE 
    text=<<TEXT 
    param=<<PARAM 
    another=<<ANOTHER
}%
one
WHERE
2
TEXT
true
PARAM
on
ANOTHER

and then reorder the TML to the same order as the docco they're reading

%ADDTOZONE{
    <<WHERE 
    text=<<TEXT 
    another=<<ANOTHER
    param=<<PARAM 
}%
one
WHERE
2
TEXT
true
PARAM
on
ANOTHER

now I'm not sure - does ANOTHER start at the end of TEXT, or... ok, back to reading the docco

next up, someone realises that the param setting is actually the default, so can be removed..

%ADDTOZONE{
    <<WHERE 
    text=<<TEXT 
    another=<<ANOTHER
}%
one
WHERE
2
TEXT
true
PARAM
on
ANOTHER

and now ANOTHER makes sense again, except of course that our poor user forgot to remove it, and so ANOTHER is affected.

similar, but slightly different result from what you get from the unnamed sections example i give below..

but as a twist - change PARAM and ANOTHER to be inputs to includetopic and excludetopic in SEARCH - suddenly the values are concatenated - interesting effect - and as its extracting the HERE's as early as possible, and they're intentionally just plain text, I wonder how we can give the user feedback so they can debug it..

Summary

To my reading, the proposal spends allot of time avoiding pointing out that its a shorthand for an existent feature :(. Its also written as though it was the solution. Its taken me until yesterday to get confirmation from Crawford/Michael and others that this in fact was only a syntactic sugar feature - so now I finally feel sure enough to think about different syntax solutions.

the thing about HERE-documents is that they take everything as verbatim up until the closing string (Crawford) is interesting - that certainly explains why you still have to escape things, unlike the longhand of using sections.

MichaelD does a pretty good job at showing how the syntactic sugar will be of benefit.

@MichaelT - 2) they do have a name 'HERE', 'END' etc are all names -

Suggestions

Obviously, a distillation of the problem space into the motivation would have made my attempts at analysis easier and faster.

• Lars suggested a variation on $here(DING) some text and new lines $here(DING) - which is pretty much the same as using $include(section='here')

As I want to start building from where we are syntactically Here-documents don't eliminate $dollar completely has an example, which can be rewritten using the sectional code

including the following not yet existent features suggestions that have come up along the way...

• $include() proposal
• add an escape="quotes/all/html/whatever?" param to INCLUDE
• OR add escape="..." param to the STARTSECTION
• if you specify INCLUDE{section="name"} then it assumes BASETOPIC
• STARTSECTION{render="hidden" - Arthur proposed it or similar

%SEARCH{ 
   "culture" 
   format="$include(section='here' topic='$topic')" 
   nosearch="on" 
   nototal="on" 
}% 

%STARTSECTION{"here" render="hidden"}%
   * %topic is referenced by: 
      * %SEARCH{ "%topic%" format="$topic" nosearch="on" nototal="on" separator=", " }%
%ENDSECTION{"here"}%

a full example?

Michael's&Crawford's example (plus and embedded Macro, and replacing the section param for fairness):

%ADDTOZONE{"body" text="%INCLUDE{"%WEB.%TOPIC%" section="initjs"}%"}%
%STARTSECTION{"initjs" render="hidden"}%
<script type=\"text/javascript\">
jQuery(function() {
  alert(\"hello world\");
});
</script>
%ENDSECTION{"initjs"}%

(10 lines, 263 chars)

would become

%ADDTOZONE{"body" text=<<HERE}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world from %TOPIC%");
});
</script>
HERE

(9 lines, 175 chars)

as a first attempt at the longhand that we will later sugar up

%ADDTOZONE{"body" text="%INCLUDE{section="initjs" escape="quote"}%"}%
%STARTSECTION{"initjs" render="hidden"}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world from %TOPIC%");
});
</script>
%ENDSECTION{"initjs"}%

(10 lines, 272 chars)

and in an attempt to reduce typing we add the following:

1. an unnamed ENDSECTION closes the closest STARTSECTION
2. Invent a shortcut to the INCLUDE - we'll call it HERE

%ADDTOZONE{"body" text="%HERE{initjs}%"}%
%STARTSECTION{"initjs" render="hidden"}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world from %TOPIC%");
});
</script>
%ENDSECTION%

(10 lines, 234 chars)

then we remember that MichaelT suggested that one nice thing about HERE is that the sections aren't named... which reminds me that STARTSECTION also works with implicit names..

and so I can suggest the possibility of something akin to (yes, ok, so it would mean that multiple SECTIONS would have to be in the order they are used)

• each unnamed use of the HERE macro would use the 'next' unnamed SECTION, and also do our quote escaping - dwim i hope?

%ADDTOZONE{"body" text="%HERE{}%"}%
%STARTSECTION{render="hidden"}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world from %TOPIC%");
});
</script>
%ENDSECTION%

(10 lines, 219 chars)

I'm sure that those that prefer less typing can find ways to reduce the typing further - I may go further later..

• can I use STARTSECTION{hidden} - no value but its a boolean that is set? (9 chars)
• HERE being a stack sounds interesting :/ (2 chars)
• what about START and END - (14 chars)
• that'd be 196 chars, only 24 to go, and thats without adding a new style of syntax - and only spending an hour thinking it.

I'm guessing that I've added 5 or so specific reasons that I'm specifically concerned about?

-- SvenDowideit - 17 Feb 2010

Thanks for the detail, Sven. I had not understood when $include() should be processed. I think I can now see how it would work. $include() would be processed after the macro handler is called, whereas the here-document content is passed to the macro, which is why, as you wrote: "you still have to escape things, unlike the longhand of using sections". (Your %HERE{}% proposal would require the same amount of escaping as here-documents.)

I agree that the proposal could use some refactoring and clarification. I agree that the proposal should address how WysiwygPlugin would treat the here-documents. I had assumed that the here-document content should be treated as if it were not a parameter (i.e. not protected), but the end-marker should be protected.

Your code-rot example in I wonder if this is pretty much like non-type checked, un-named parameters is interesting. I value debuggability highly, so IMO that is a strong argument. The problem could be addressed with a macro that escapes its parameters and then outputs them e.g. in a verbatim block (easily implemented as a plugin - maybe I should write that anyway )

Yes - "HERE" and "END" are names. However, they differ in scope from section names. Here-document end markers are "more local", by which I mean that I may use the same name for the end marker several times within one topic without fear of confusion. Of course, it is possible to use here-document end markers in a way that is very confusing indeed, but it is possible to write bad code in any language (just like overlapping sections is powerful but confusing).

You counted characters - that is not the metric I would use. I would rather look at the number of discrete things being used together. Experienced foswiki app-writers "chunk" the use of %SECTION and %INCLUDE, making section-include a single idea. However, newish users (such as myself) treat those as separate things.

So now there is: (1) include, (2) of-a-section (as opposed to a plain include), (3) start-of-section, (4) end-of-section and (5) hide-the section. By comparison, I would say that using a here-document would involve (1) use-here-document-quoting and (2) choose-a-name-for-end-marker. I am not an expert in psychology or in training, but I can see in my own field that some tasks are incredibly hard to learn if there are many new concepts to learn and apply at once. The current documentation for %SECTION and %INCLUDE is fragmented, so to a new user, including a hidden section really does involve many concepts which are not related to what the user was trying to do in the first place. IMO, the here-document syntax would be easier to learn.

Sure, the documentation can be improved. I think it should be improved, even if that is the only thing that comes out of this proposal.

I like the here-document syntax. But then, I implemented it, so I may well be biased . Others had mentioned they didn't like it. Do they still not like it?

-- MichaelTempest - 18 Feb 2010

While I was minding the girls today, I had another idea

the desired goal is to simplify Macro parameters specifically whereas I use sectional includes more generally.

thus, I realized that we don't need two names for each param (both text and HERE are names, and quite duplicate..)

so we could remove the quoting by making an in Macro parameterlist shortcut... called P(someparam) (for argument's sake) which would expand to someparam="%!INCLUDE{"%WEB.TOPIC" section="_SECTION0"}% (making continued use of the un-named sections)

%ADDTOZONE{"body" P(text)}%
%STARTSECTION{render="hidden"}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world from %TOPIC%");
});
</script>
%ENDSECTION%

(10 lines, 211 chars)

and now, we're at a point where we could talk about the %!STARTSECTION{render="hidden"}% being implicit, and even putting the param name into the =ENDSECTION

%ADDTOZONE{"body" P(text)}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world from %TOPIC%");
});
</script>
%ENDSECTION{text}%

(9 lines, 185 chars) - or 178 chars using END

cute thing is that I would still be left with a SECTION that I could transclude from another topic - thus increasing the possibility for unplanned reuse.

What I'm doing with these examples, is showing a way to get to a similar shortcut, without the need for another extraction method - instead, it is achieved via the more typical syntax sugar method of pre-processing the shortcuts, and replacing them with the longhand versions.

wrt psychology or training - good point, I taught C and C++ to engineers and research PhD's, and Cicode (custom development language for Citect) to engineers and Electricians, and the most relevant thing I recal, is that simplistic consistency matters the most. Most pain is caused when there is a different syntax style for something that could actually be achieved using the main syntax - thus my reflexive and ill defined dislike for the <<HERE syntax in October.

Have I provided enough examples of an alternate syntax to allow you to think up more?

-- SvenDowideit - 18 Feb 2010

OK, now I feel really stupid. The main motivator for this proposal for me was wanting to be able to format structured statements more cleanly. For example, I want to write something like:

%IF{"$idiot='yes'"
then="
   | %WIKIUSERNAME% | idiot |
" else="
   | %WIKIUSERNAME% |
   | genius |
"}%

Somewhere along the line I didn't notice that in the course of various parser changes, the macros parser handles newlines inside macro params just fine - try it (https://foswiki.org/Development/HereDocumentSyntaxForMacros?idiot=yes#WhatAnIdiot to flip the condition)

WikiGuest

genius

MichaelTempest pointed this out above, but I assumed he meant newlines between parameters. So, right now I'm thinking we don't need this proposal at all, and should focus instead on enabling late eval. That's not to take anything away from the useful discussion on the merits of sectional transclusion.

Of course the doc needs to be clearer on this; and of course it doesn't apply to plugins that use commonTagsHandler, and of course it's not the case for %SECTION or %TOC which are differently abled.

-- CrawfordCurrie - 18 Feb 2010

Here-documents might not be needed for newlines in parameters given that inline parameters already cope with that. However, here-documents do also address quotes in parameters, which require escaping in inline parameters. How should this need be addressed, if not with here-documents? That is not a rhetorical question - I believe it remains an open issue if this proposal is rejected in its current form, as seems likely at this stage.

-- MichaelTempest - 19 Feb 2010

I just want to cheer up Michael and everybody else working on this proposal and say I'd really love to see HERE documents in some mid-distant future. Remember, this is a feature targeted at one of our next production releases. However, wiki apps won't start making use of this feature as soon as HERE documents materialize because lots of these apps still have to run on foswiki engines that don't have HERE documents.

So again, cheer up, this needs a bit of breath to (a) get accepted in a reasonable form and (b) see being used in the wild.

My Sunday 2cent.

-- MichaelDaum - 21 Feb 2010

@Crawford - I had here-documents working for %SECTION, in distro:bb0e75a0ef78.

I think I will remove the current implementation of here-documents soon, because I implemented it so that we could play with it, understanding that it would be removed if the proposal was not accepted. In case we want to come back to it later, distro:9768ca512d41 was the last Tasks.Item8422 checkin for the original implementation, corresponding to the original specification for here-documents.

That will give space for considering a different syntax.

-- MichaelTempest - 21 Feb 2010

Regrettably I think that would be the right decision at this stage. I played with your impl, and still much prefer it to the $include proposal, but the time is not ripe. Somewhere along the line we lost focus on the other aspect of the original proposal, viz. late eval, and I'd like to revisit that. BTW, do we have any tests for newlines in the double-quoted params to %SECTION?

-- CrawfordCurrie - 22 Feb 2010

We do not have any tests for newlines in double-quoted params to %SECTION.

-- MichaelTempest - 22 Feb 2010

I reverted the here-document implementation, and I removed the date of commitment, since we don't have a viable spec yet.

Sven - I like the P(text) }% ... %END{text}% markup. We should consider how it would be affected by UseSyntaxToChangeEvaluationOrder.

I am concerned that the implicit section-start could slow down section processing. The STARTSECTION/ENDSECTION code would have to check each macro's parameters to see if they start a section, and that would mean changing the section parser to be more like the macro parser. We could avoid this if we decided to forgo the idea that P(text) }% ... %END{text}% defines a section.

Also - why P? Why not simply (text)? Could we define processing that happens based on the name before the parentheses?

-- MichaelTempest - 23 Feb 2010

@Crawford - um, i'm not talking about $include here. please read my actual propsal.

I've added the related version of my proposal to the UseSyntaxToChangeEvaluationOrder partially to show why I think the two proposals can and should be considered seperatly, and partially because I see it as a nobrainer extension of the existing format work that lead me to consider $include() for it.

I don't think the implicit STARTSECTION sill cause section processing to slow down, specifically because I am proposing that we consider 'shortcut' syntactic sugar to be a pre-processing step - at least until we work out an improved parser.

P - I am of the opinion that we should not do something like (label) as that's much more implicit and de-readable than being more explicit. however, I chose a letter at random - personally, i would prefer a word.

also, I think I would like to ammend that idea to be (again, INC chosen for no good reason - I'm poor at choosing colours and letters)

%ADDTOZONE{"body" %INC{label}%}%
<script type="text/javascript">
jQuery(function() {
  alert("hello world from %TOPIC%");
});
</script>
%ENDSECTION{label}%

-- SvenDowideit - 24 Feb 2010

Forgive me, but it looks to me like you just re-invented heredocs, with a subtly different (though still new) syntax.

• %INC{label} ← label=<<LABEL
• %ENDSECTION{label}% ← LABEL

Or am I missing something?

-- CrawfordCurrie - 25 Feb 2010

I never complained about the ideal, just that the syntax is un-necessarily inconsistent, and the docco implied that this feature is much more complex than in fact it turns out it is.

and as you complained that i had not provided an alternative, I presented you with one.

there are differences between my suggestion and yours, and quite useful ones - but the important point is that i don't feel that you did enough analysis before you presented something as a 'solution'

-- SvenDowideit - 25 Feb 2010

One very useful difference between label=<<LABEL and %INC{label}% is that I can pass parameters to %INC e.g. =%INC{format outertopic="$topic"}%

-- MichaelTempest - 25 Feb 2010

Changing to ParkedProposal. Developer has left project, and this proposal has concerns.

-- Main.GeorgeClark - 03 Nov 2015 - 16:24