Feature Proposal: Setting Variables Using Macros

Motivation

The standard syntax for setting a variable is
* Set FOO = bar

However this prevents them to be set as part of a wiki application. A
%SET{"FOO" value="bar"}%

could be the natural Foswiki way to do the same.

The big difference between Set FOO and =

This macro is used to set a temporary value for the named preference. This temporary value is only set when the topic is prepared for viewing, and exists from the time the %SET macro is expanded through to the end of the current scope. This is either:
  1. The end of the topic being viewed, or
  2. The end of the topic in which %SET is used, when the %INCLUDE macro is used to include it.

The %SET macro doesn't generate any text; its only effect is to set the preference, which can then be used like any other preference.

Tests

%IF{"defined foo" then="FAIL" else="SUCCESS"}%

%IF{"isempty foo" then="SUCCESS" else="FAIL"}%

%SET{"foo" value="bar"}%

foo=%foo% (should be bar)

%IF{"defined foo" then="SUCCESS" else="FAIL"}%

%IF{"isempty foo" then="FAIL" else="SUCCESS"}%

%SET{"foo" value="baz"}%

foo=%foo% (should be baz)

%SET{"foo"}%

foo=%foo% (should be "empty")

%IF{"defined foo" then="SUCCESS" else="FAIL"}%

%IF{"isempty foo" then="SUCCESS" else="FAIL"}%


<verbatim>
%STARTINCLUDE%%SET{"foo2" value="bar2"}%foo2=%foo2%%STOPINCLUDE%
</verbatim>

%INCLUDE{"%TOPIC%"}% ... (should be bar2 ... local variable)

%foo2%=%foo2% ... (should be undefined)

%IF{"defined foo2" then="FAIL" else="SUCCESS"}%

%IF{"isempty foo2" then="SUCCESS" else="FAIL"}%

tests from skin templates (for eg)
%TMPL:DEF{"simpleheader"}%%SET{"foo2" value="bar2"}%foo2=%foo2%%TMPL:END%

Community Vote

This proposal has been discussed for almost 3 years now. Concern has been raised. There is no sign of a consensus being reached.

Time for a community vote again.

The vote runs for 7 days starting 10 Aug 2012 and ending 17 Aug 2012 at midnight GMT. The code to be contributed is available as a patch part of this proposal. Unit tests have been outlined as well.

The question is: Do you agree that %SET should be added to the core?

Name Vote (Yes/No) Reason
Yes
a simple frontend to a feature available via Foswiki::Func on perl level already, that will be of a lot of value to wiki application programmers
Yes
Much simpler approach to this common task. Have had to work-around on many occasions
Yes
However it is imperative that "fix Foswiki::Func::setPreferencesValue() not to be able to change an already finalized preference variable" is done, otherwise it opens a huge hole
Yes
Very useful feature for application development. I'm assuming security matters have been worked out since Macros are used in setting permissions.
Yes
Seems to be an useful addition
Yes
I have no concerns.
Yes
Seems quite useful. I guess wide scale use is needed at this stage to really detect the implications anyways. I share GeorgeClark concern
Yes
I still feel it is geeky and not adequately explained. However a TML interface to Foswiki::Func::setPreferencesValue= does make sense.
Yes
I do want this feature, my concerns have never been addressed (I've added an overly simplistic eg above), so presumably we'll just hit them during testing / several years later
Yes
I could have used this three years ago smile
Yes
More an expert user a la app dev feature than a plain end user feature so some geekiness is OK. And we can help each other making the user docu clear. So OK with me. I personally see good use of it
Yes
 
Yes
I see fantastic use for this feature. Document types (data forms) can make type specific macros available through view- and edit-templates. I just hope it gets not to messy in regard of expansion sequence!
-- Contributors: MichaelDaum - 13 Nov 2009

Discussion

[Friday 13 November 2009] [17:21:32] <MichaelDaum> CDot, I just came across the need to deal with macro parameters of the
same name returned as an array (in perl)
[Friday 13 November 2009] [17:21:50] <MichaelDaum> like cgi params in an array context
[Friday 13 November 2009] [17:21:54] <CDot> MichaelDaum: "returned"?
[Friday 13 November 2009] [17:22:09] <MichaelDaum> the Attrs.pm overrides them ... last come wins
[Friday 13 November 2009] [17:22:11] <CDot> oic; %X%=a,b,c,d
[Friday 13 November 2009] [17:22:17] <CDot> yeah, probably
[Friday 13 November 2009] [17:22:32] <MichaelDaum> more like this: MACRO{filter="this" filter="that"}
[Friday 13 November 2009] [17:22:52] <CDot> yep. Well, Attrs.pm can easily be extended.
[Friday 13 November 2009] [17:22:53] <MichaelDaum> and then in the macroHandler: my @filter = $params->('filter');
[Friday 13 November 2009] [17:23:40] <CDot> what does %filter% mean in that context?
[Friday 13 November 2009] [17:23:42] <MichaelDaum> problem is how to deal with normal code that does an array access: $params->{filter} ...
you still need to return the last elem from the array not the array ref
[Friday 13 November 2009] [17:23:56] <MichaelDaum> filter: an arbitrary param
[Friday 13 November 2009] [17:24:13] <MichaelDaum> s/filter/random param name/
[Friday 13 November 2009] [17:24:25] <CDot> yeah, but if I say %MACRO{filter="this" filter="that"}% what is the value of %filter%?
[Friday 13 November 2009] [17:24:48] <MichaelDaum> the last in the row -> back warts compatibility
[Friday 13 November 2009] [17:26:08] <CDot> ok, so what other cases are going to bite
[Friday 13 November 2009] [17:26:20] <CDot> %SEARCH{type="query" type="keyword"}%
[Friday 13 November 2009] [17:26:26] <MichaelDaum> you actually might want to retrieve the list in wiki app space
[Friday 13 November 2009] [17:26:34] <CDot> yes
[Friday 13 November 2009] [17:26:44] <MichaelDaum> something along the lines of %@filter%
[Friday 13 November 2009] [17:26:53] * CDot throws up
[Friday 13 November 2009] [17:27:20] <CDot> %LISTPARAM{"filter"}%
[Friday 13 November 2009] [17:27:27] * MichaelDaum throws up
[Friday 13 November 2009] [17:27:48] * CDot cleans the vomit off his keyboard and tries again.... %[filter]%
[Friday 13 November 2009] [17:28:00] <MichaelDaum> actually %listparam{}% is not bad
[Friday 13 November 2009] [17:28:14] <CDot> well, it's analagous to %URLPARAM
[Friday 13 November 2009] [17:28:43] <MichaelDaum> what about %PARAM{}%
[Friday 13 November 2009] [17:28:46] <CDot> how does URLPARAM handle lists?
[Friday 13 November 2009] [17:28:58] <MichaelDaum> URLPARAM multiple="on" separator=","
[Friday 13 November 2009] [17:29:13] <MichaelDaum> URLPARAM format="$item," is broken btw
[Friday 13 November 2009] [17:29:36] <MichaelDaum> does return thisthat, instead of this,that,
[Friday 13 November 2009] [17:30:04] <CDot> so %PARAM{"filter" multiple="on" format="$item" separator=","}%
[Friday 13 November 2009] [17:30:18] <MichaelDaum> so %PARAM{"filter" multiple="on" }% would be the proper analogy
[Friday 13 November 2009] [17:30:25] <MichaelDaum> snap
[Friday 13 November 2009] [17:30:27] <CDot> guess so
[Friday 13 November 2009] [17:31:00] <CDot> question is, if you know you want the second one, how do you refer to it?
[Friday 13 November 2009] [17:31:18] <CDot> we have no way of representing, or handling, a list value
[Friday 13 November 2009] [17:31:22] <MichaelDaum> %PARAM{"filter" multiple="on" skip="10" limit="5"}% 
[Friday 13 November 2009] [17:31:32] <CDot> * Set VALUE = scalars only, chaps
[Friday 13 November 2009] [17:31:53] <CDot> * Set @VALUE = [ 1, 2, 3 ]
[Friday 13 November 2009] [17:32:18] <CDot> * Set %VALUE = { a => b, c => d }
[Friday 13 November 2009] [17:32:18] <MichaelDaum> what about %SOMEVAR{value="1" value="2" value="3"}%
[Friday 13 November 2009] [17:32:28] <CDot> shoot me, someone, before I re-invent perl
[Friday 13 November 2009] [17:33:06] <MichaelDaum> %SET{name="filter" value="this" value="that"}%
[Friday 13 November 2009] [17:33:22] <CDot> hmmm.
[Friday 13 November 2009] [17:34:14] <MichaelDaum> the counter part would be a %GET
[Friday 13 November 2009] [17:34:21] <MichaelDaum> instead of a %PARAM
[Friday 13 November 2009] [17:34:29] <CDot> well, it would be consistent. %SET{"filter" value="this" value="%filter%"}%
[Friday 13 November 2009] [17:35:06] <CDot> make %GET handle urlparams as well, and I could learn to like it.
[Friday 13 November 2009] [17:35:06] <MichaelDaum> %SET has got some nice advantages over * Set =
[Friday 13 November 2009] [17:35:20] <CDot> %SET has *huge* advantages, IMHO
[Friday 13 November 2009] [17:36:32] <MichaelDaum> let me copy-paste this to a proposal
<CDot>   oh, also, %SET cannot affect permissions settings. So the terminology, I guess, is "Preferences" (where * Set used to set default values) and "Macros" where preferences provide default values, but those values may be overridden in %SET to provide a new value that can only be referenced using %GET) sounds dangerously like SpreadShitPlugin    [17:20]
<MichaelDaum>   naw no new namespace they go to preference land. not sure if acls are stored there actually    [17:21]
<CDot>   also, if JoePluginAuthor writes getPreferenceValue("MUNGE") they will retrieve * Set MUNGE but not %SET{"MUNGE" which could be confusing    [17:23]
<MichaelDaum>   should be avoided my @list = getPreferenceValue retrives the list ... in scalar context it is the last list elem
acls themselves are lists    [17:25]
<CDot>   erm, you can't have getPreferencesValue return the result of a %SET
it *must* return a static preference; otherwise the value is variant over the different handlers, isn't it?    [17:26]
<MichaelDaum>   wantarray?    [17:27]
<CDot>   * Set preferences are still scalar values    [17:27]
<MichaelDaum>   oic    [17:27]
<CDot>   if you want an array from it, you should have to split(',', Foswiki::Func::getPreferencesValue("MUNGE"))    [17:28]
<MichaelDaum>   which is too fragile    [17:28]
<CDot>   very fragile
Foswiki::Func::makeList($scalar) -> @list    [17:28]
<MichaelDaum>   so can Func::setPreferenceValue override ACLs?
if not then %SET has good changes not to either
<CDot>   on setPreferenceValue; not sure about that.... checking
No. setPreferencesValue sets a SESSION level preference. Session level is not checked for ACLs.    [17:45]
<MichaelDaum>   okay so session level is the natural place where all %SET goes?    [17:47]
<CDot>   I guess so
though it's still variant over the duration of the session, which worries me slightly
could be a devil to debug

-- MichaelDaum - 13 Nov 2009

This is a very important proposal. For instance if TINYMCEPLUGIN_INIT and WYSIWYGPLUGIN_STICKYBITS hashes, it would be much, much easier to override and adjust only parts of such large configuration variables.

I hope we can also think about how list types in metadata might be implemented.

I am having to do very, very ugly work to mimic hashes:

Field name Hash equivalent
Foo_Bar
Foo['Bar']
Foo_Bar_Tree
Foo['Bar']['Tree']
FormFieldListPlugin provides a really tedious way to extract individual and groups of fields that I need.

-- PaulHarvey - 13 Nov 2009

Be very, very careful to understand the implications of this.

A %SET makes a macro value variant over the lifetime of a request. Preferences are currently invariant - once set, they do not change. To understand what this means, consider this scenario. For illustration purposes, let's say our topic contains a * Set X=1 in the topic body, == (proposed) We're getting carried away. Let's reset a minute, and examine our real requirements. We want:

  1. to maintain compatibility with existing SEARCH, META and FORMFIELD macros
  2. to be able to debug where a preference is defined
  3. to be able to retrieve the value of a meta-datum
  4. to be able to retrieve the value of a meta-datum in a different topic
  5. to be able to retrieve the raw value of a preference in the current scope
  6. to be able to retrieve the raw value of a preference in a different scope (another topic)
  7. to be able to set the raw value of a preference
  8. to be able to post-process the value of a preference for display (e.g. s/\n/
    /g)
  9. to be able to post-process the value of a meta-datum for display

If I missed any, please add them.

Now we have a bunch of implementations and proposals that address some of these requirements (Y = full support, N = no support, though you might have expected it, , p = partial (subset) support, blank = no applicable)

Macro 1 2 3 4 5 6 7 8 9
META
Y
 
p
N
 
 
 
 
p
FORMFIELD
Y
 
 
p
p
 
 
 
p
SEARCH
Y
 
p
p
p
p
p
Y
Y
SHOWPREFERENCE
Y
Y
 
 
 
 
 
 
 
QUERY
Y
 
Y
Y
Y
N
 
 
 
SET
Y
 
 
 
 
 
Y
 
 
GET
Y
 
 
 
Y
N
 
Y
 
EXPAND
Y
 
 
 
Y
Y
 
N
 
FOREACH
Y
 
 
 
 
 
 
Y
Y
If I missed any, please add them. There seems to be a lot of overlap and a holes in the preference getting support, and in the value formatting.

Michael, as I understand it, Sven is saying that some combinations of these macros can simplify this for us (ignoring META, FORMFIELD and SEARCH), we can combine rows:
Macro 3 4 5 6 8 9
QUERY + FOREACH
Y
Y
Y
 
Y
Y
GET + EXPAND + FOREACH
 
 
Y
Y
Y
Y
Sven, in this proposal, Michael is (by implication) putting forward the case that all macros that recover preferences/meta should support the format header etc. params. So, the conflict I see here (if there is one) is between there two points of view - post processing for formatting, or formatting in the core macro.

Personally I see the FOREACH approach as the cleanest and most general. However I have concerns about the inside-out-left-right order of evaluation, and the use of comma-separated lists:
  1. What happens when the value (preference or meta) contains a macro call?
  2. What happens when a value contains a comma?
So pragmatically, Michael's approach appears to be the low hanging fruit. It also in principle deals with some of the corner cases that FOREACH misses, such as newline, though there doesn't appear to be any shareable code that implements such features.

We need some serious thought on this. I'm going to spend a few hours today experimenting with FOREACH to see where the holes are (if there are any)

-- CrawfordCurrie - 26 Feb 2010

You're still missing what the extractFormat feature is all about. FOREACH is only one way to access the formating code. The next part of that refactor - which is scheduled for 2.0, is to use that same code as a Foswiki::Func and internally for all uses of format, header, sort, paging etc.

not just Macros that recover preferences, but all macros that have output.

What I'd like to do, is have Michael use and extend the existing code which is destined to become the one Macros formatting system, rather than duplicating, and making something that is subtly different from it.

I really don't think we should force our users to learn 2 'master' macros to access TOM addressable elements, nor do I beleive that its to our advantage to add another format codebase.

(please make unit tests fo your experimentation smile )

-- SvenDowideit - 27 Feb 2010

First of all: I am not integrating SetVariablePlugin with this work. SET is not meant to provide persistence. It is meant to be a TML conform way of saying <space><space><space>* Set FOO = BAR. If at all this is comparable with %CALC{"$SET()"}%, which does establish a namespace of variables completely separate from preference variables. %CALC in itself is a beast as its evaluation order conflicts with the rest of the TML language in many places, thus rendering it a PITA to get sorted out, with users working around evaluation artifacts more than getting real work done.

Second: there is a severe limitation in Foswiki::Attrs only allowing one value for a parameter of the same name. For example, the value of param in %INCLUDE{"topic" param="bla" param="blub"}% is not specified formally. It only happens to be one of the two (dunno by heart which one, i guess the last one wins). Having params of the same name to a parametrized function call is however a missing feature when you'd like to pass lists. Today, you'd format a list value as comma separated list and then separate the list again in the body of the called function. That's awful.

That's why I'd like to extend Foswiki::Attrs to capture all values of param in a list instead of overriding the last definition. Overriding the former value bla with blub would still work as is for backwards compatibility.

Now, to retrieve all values of param, we need a kind of access function. I proposed GET for that.

Note, that the nature of SET and GET is

  • to only operate on the namespace of preference variables, not meta data of any other kind

and

  • to modify variables on "session" level, that is not on user, web, plugin, or topic level.

The required functions are all available in the Foswiki::Func API already, that is setSessionValue() and getPreferenceValue().

So far a straight-forward quite simple contribution to the core.

Both EXPAND and QUERY weren't on the radar when I proposed this.

A similar extension to Attrs has been proposed on the other project recently.

-- MichaelDaum - 27 Feb 2010

OK, that's what I understood. I appreciate that EXPAND and QUERY weren't on the radar, and I don't actually think QUERY is relevent to this discussion (except where it impinges on the FOREACH debate). EXPAND fulfils a different function, to expand a macro (not just a preference) in variable scope.

So, the difference between %GET{"BLAH"}% and %BLAH% is the ability to format the result of the GET as a list? So we come back to the debate regarding a functional FOREACH approach versus format support in individual macros. Can you comment on this please?

With regard to lists; I don't have a problem with extending Attrs to support multiple values. Note that perl semantics when assigning a list to a scalar is for the last value in the list to be taken as the scalar value. I assume we are talking here about the scalar value of a list value being the join(',',@list), which raises the spectre of commas in values again. Preference values are stored as scalars, so this is the same issue as bites FOREACH, and I don't see that GET is any better that FOREACH in this respect. How do you propose to deal with this?

-- CrawfordCurrie - 28 Feb 2010

From what I know about FOREACH so far, it tries to unify formatting by proposing constructions like this:


%FOREACH{
   "%INNERMACRO%" 
  format="..." 
  header="..." 
  footer="..." 
  separator="..."
}%

%INNERMACRO% must format its list result separating it using comas to please %FOREACH%.

(BTW: a list like "orange , apples" fails to split the list in a robust way; there's no way to define a split pattern or any other means to decompose the result of INNERMACRO into a proper list. There are other shocking weirdnesses in the code. Let's put that all aside for a moment as I don't want to discuss this here.)

In general, the way INNERMACRO and FOREACH pass around the list - that is via text substitutions on the wiki application level - is horrendously inefficient.

Because:

  • (1) the INNERMACRO presumably already has got some sort of perl structure representing a list
  • (2) it then stringifies it to a list to return its result
  • (3) finally FOREACH parses this string and converts it back to a list for the purpose to reformat the result in a different way.

Fragile, error prone, inefficient.

INNERMACRO should use internal APIs to format its list results right away using the FFs.

Even better: instead of communicating a list using an inefficient stringification and decomposing cycle in the middle, the internal list object is bound to a %VARNAME% (in session layer) and GET (or FORMAT) can access it directly using its name as an id.
%GET{"VARNAME" <- not %VARNAME% !!!
  header="..."
  format="..."
  separator="..."
  footer="..."
}%

A plain %VARNAME% - without a GET would expand to a comma separated list for simplicity given it is a list object.

-- MichaelDaum - 28 Feb 2010

Very good points, and your observations about FOREACH are accurate, but you have just moved the list parsing problem back a stage. %GET still has to recognise where a preference represents a list. At the moment if I define:
  • Set BLAH = blah , blah ,blah
Then I get a preference with a value "blah , blah ,blah". What should I expect if I %GET{"BLAH" format="$item" separator=","}% (or, for that matter, %FOREACH{"%BLAH%" format="$item" separator=","}%? How do I express a list that contains a value that contains a comma?

Sven has proposed standardising the code that handles format, header and I full support that idea. So we can reasonably assume that %GET and %FOREACH will end up using the same code to generate their output. The question remains, though, as to what code they use to parse their input.

-- CrawfordCurrie - 28 Feb 2010

Right. But the point is to prevent having to parse the input back and forth. I want MACROS to talk to each other more directly and process result sets internally, instead of going through the TML parser.

Now, there are some lists that do need to be parsed. FOREACH does not provide sufficient means to do so. FORMATLIST of FilterPlugin does and has been proven to be very flexible for a lot of different kinds of lists. However, it lacks storing these lists in a shared namespace for post processing.

But, there are also some lists that better are not stringified and remain lists internally, represented by a result set ID. We should work on specifying this as a means to process these lists repeated times.

Crawford, you've done something of that kind in FormQueryPlugin. SolrPlugin does it as well. So it is time to get a good result set framework in place.

-- MichaelDaum - 28 Feb 2010

We all want lists handled internally, and passed between macros and stored in preferences. But we are quite a long way from that. Until we are a lot closer, we are faced with the question of "what is the string representation of a list?", and that is the key *question I'm trying to get you to help answer. Right now, by ignoring/fudging around the question, we are faced with lists that are defined as: split( /\s*,\s*/ ) - which of course precludes the use of commas in tokens. Maybe that's sufficient; if it is, we need to standardise on it, in the same way as we standardised on the formatting tokens.

-- CrawfordCurrie - 28 Feb 2010

Only allowing lists to that simple kind is very limiting.

Let's recap the docu of FORMATLIST from FilterPlugin as that's the most advanced list parser that we have atm.

(Not that I think that this is a discussion that belongs here. However you asked for not "fudging around the question". So here you are.)
%FORMATLIST{
   "some list"
   <list parser arguments>
   <list filter arguments>
   <list formatting arguments>
}%

List parser arguments are:

  • tokenize="...": regex to tokenize the list before spliting it up, tokens are inserted back again after the split stage has been passed
  • split="...": the split expression (default ",")

List post processing and filter arguments

  • exclude="...": remove list items that match this regular expression
  • include="...": remove list items that don't match this regular expression
  • limit="...": max number of items to be taken out of the list (default "-1")
  • skip="...": number of list items to skip, not adding them to the result
  • sort="on,off,alpha,num,nocase" order of the formatted items (default "off")
  • reverse="on,off": reverse the sortion of the list
  • unique="on,off": remove dupplicates from the list

List formatting arguments:

  • pattern="...": pattern applied to each item (default "\s(.*)\s")
  • format="...": the format string for each item (default "$1")
  • header="...": header string
  • footer="...": footer string
  • separator="...": string to be inserted between list items
  • selection="...": regular expression that a list item must match to be "selected"; if this matches the $marker is inserted
  • marker="...": string to be inserted when the selection regex matches; this will be inserted at the position $marker as

Here's what happens inside:

  1. list construction phase
    1. all standard escapes in the list are replaced and the result is executed (expandCommonVariables)
    2. the list is processed using the tokenize parameter by temporarily removing matching substrings before proceeding to the next step
    3. the list is split according to the split pattern
    4. tokens gathered in step (2) are inserted back into the list
  2. list processing phase
    1. the list is sorted according to the sort parameters
    2. if specified its order is reverted
    3. any unwanted list items not matching include or matching exclude are removed from the list
    4. if unique was switched on, duplicate items are removed.
    5. the list is reduced to the specified sublist as specified by skip and limit
  3. output phase
    1. for each list item pattern is matched and grouping is applied as specified in the pattern parameter.
    2. each item is formatted using the format parameter and pushed onto a result stack; groups caputred via pattern are inserted via $1, $2, ...
    3. if a list item matches the selection expression marker is inserted (we've got that in WEBLIST as well; avaliable here for more general purposes)
    4. the result stuck is joined using the join parameter and a header and footer is added

Here's a more interesting list and a best-practice for using FORMATLIST as well:
%FORMATLIST{"
    "oranges=2;apples=15;grapes=1,5"
   split="\s*;\s*"
   pattern="(.*)=(.*)"
   format="   * $1 is $2"
   separator="$n"
   footer="$n found $count items"
}%

There are a lot of situations where you can encode lists this way with items consisting of tupples of information bits that belong together.

Another interesting thing to work around list separators that might occur in the list items as well is the tokenize parameter. This can be used to sort of protect items from being misinterpreted as separate list items.

What would make a helluffa sense is to split out a PARSELIST from FORMATLIST and add an id parameter to both. So PARSELIST could store a list under the given id and multiple FORMATLISTs could format it in various ways, multiple times.

In fact, I'd like to see is namespace of IDs being shared among all macros. Behind a specific ID we have a result object with a standard interface to iterate over all contained items. For ease of use we could use TIEs for that. But that's already too impl specific. Let's stop this here and see if we can formally define ResultSets.

One central design factor is to hide lazy retrieval inside a result set, that is: never ever fetch all of the result set in advance trying to load it all into memory. That's what SEARCH in trunk currently does - a cardinal flaw.

-- MichaelDaum - 28 Feb 2010

We have talked about result sets before in SearchResultsPagination.

-- ArthurClemens - 28 Feb 2010

That's only covering the output phase of some search specific result structure.

-- MichaelDaum - 28 Feb 2010

I agree that that topic shows a limited view of what has been in our heads, and you have summarized it above:
  • Querying involves handling lists, or collections (ordered or unordered)
  • Collections involve a result set, then a sort step, a filter step and a display step
  • Result sets should be stored (memory/db/disk) for quick retrieval or to manipulate them

-- ArthurClemens - 28 Feb 2010

After re-reading the discussion here it has wandered so far from the original proposal that I think the "accepted" status needs to be withdrawn and the proposal revisited. Set back to investigation state until the outstanding concern can be removed (it can't, from my reading of it)

-- CrawfordCurrie - 24 Feb 2012

Implementing %SET as specified is as trivial as interfacing Foswiki::Func::setPreferencesValue().

Stripped down the proposal to that.

-- MichaelDaum - 20 Apr 2012

I never thought it would be hard to implement - things like FormaliseTheProgrammingLanguage just make me wonder if its the right syntax to implement.

its pretty scary that given a few minutes hacking, I was able to implement the same result using TML only.

So - the proposal is missing the motivation - Why do we want to confuse users with preferences that look the same as the static pre-rendering extracted values, which change randomly during the render phase? (its worth answering in detail)

I'm leaning towards %SET at the moment - but don't expect me to jump for joy until i play some more with the other options.

so - in the context of Crawford's scope proposal - what scope is %SET? do we add a scope param to %SET to push up to a topic's scope (dangerous, but I thought a large part of what you wanted was a global %SET, not a local one)

-- SvenDowideit - 20 Apr 2012

The scope question is key; and it's that that makes me wonder if SET or LOCAL are the right name. Let's see, we're trying to say "here's a temporary preference that will stay active until it drifts out of scope". So maybe %TEMPORARY (or %TMP for brevity) would be more apt? Or - given that the world is driven by perl - %MY{"foo" value="bah"}%

A feature proposal like this needs user documentation, so here's my stab at the doc (moved to top of proposal)

-- CrawfordCurrie - 20 Apr 2012

%MY makes sense coming from perl. Doesn't make sense otherwise cus: why the f* is this called "my", this is not "my" variable. Fankly the "my" and "our" labels don't make sense in perl either.

The "my" meme is all over the (social) internet, like in "my contributions", "my settings". So a "%MY{"color" value="blue"}%=" might be the wrong message here as users might read this as "my color is blue" rather than "set this color to blue in the current scope".

Imho, %SET is okay as it does what it says. It even is called like this in Foswiki::Func.

Scope is an interesting - and tricky - thing. For now we only have local scope which is nice for a lot of use cases. However there are lots of useful things to have global scope for, i.e. to have sideeffects in totally decoupled places. For instance: use %SET to change things in the sidebar navigation. There are some tricky parts, most of them have to do with evaluation order in the skin itself: when is the sidebar computed and when is the topic text rendered, which could become very confusing.

-- MichaelDaum - 20 Apr 2012

The problem with SET is that it implies permanance. A * Set doesn't just apply from the position of the set onwards; it applies throughout the topic. A *Set also has different semantics in included topic. So *Set and SET are not the same, so should not share the same name. As an exercise, try to explain it to a noob on IRC.

-- CrawfordCurrie - 20 Apr 2012

I did smile
(16:19:58) total_noob: MichaelDaum: can you explain to me the difference between %SET and * Set please? I'm confused......
(16:25:30) MichaelDaum: total_noob, the normal * Set isn't processed as part of parsing the TML language. it is done on a separate stage _before_
(16:25:51) MichaelDaum: by reading WebPreferences, and SitePreferences and wherever they hide themselves
(16:26:14) MichaelDaum: this then forms the current context within which all TML is then evaluated
(16:26:43) MichaelDaum: now, %SET is a macro as any other %MACRO being parsed as part of the TML language
(16:27:12) total_noob: why do they have the same name, then?
(16:27:23) MichaelDaum: so there comes the big differences: the preference variables can be in the linear parsing order of any other TML macro
(16:27:52) MichaelDaum: they have the same name as they set the same variable in the same namespace of variables
(16:28:13) ***total_noob has a brain meltdown when he reads "linear parsing order"
(16:28:31) ***total_noob doesn't know what a namespace is
(16:29:12) MichaelDaum: each variable is known by its name.
(16:29:22) MichaelDaum: this name is the key to the value of the variable
(16:30:41) MichaelDaum: so I guess you see the difference now between * Set and %SET don't you?
(16:31:42) total_noob: I do, but total_noob is utterly confused; he doesn't know what "linear parsing order" and "namespace" mean.
(16:31:47) MichaelDaum: linear parsing order means: there is a deterministic order in which program code is evaluated. there is no guessing in that.
(16:32:34) MichaelDaum: all deterministic things can be linearized. thats why I said that.
(16:32:37) ***total_noob thinks that *Set should be *Constant, and %SET is the correct name for that
(16:33:25) total_noob: as in, it's not the new macro %SET that is mis-named; it's the original *Set that is wrong.
(16:33:28) MichaelDaum: total_noob, dont try to split hair. sometimes you'll be a happier noob letting all hair - that you still have - grow out of your ears.
(16:34:01) ***total_noob is careful not to sit too close to the fire, in case his ear-hair catches fire

-- MichaelDaum - 20 Apr 2012

Yup, that was a fail.

  • %MY won't work - its Perl-ish
  • %SET won't work - users are already confused by the word Set - and we really don't want to make another mess like we have with the word template

what we need, is something that makes the ephemeral-ness of this name-value association.

I'm presuming that %ELPH will work throughout the entire (single) request - including templates - but that for jqLoader delays, it'd have to be passed as a param

-- SvenDowideit - 22 Apr 2012

%ELPH? makes as much sense as %TROLL or %WYVERN, I suppose.

I suggested %TEMPORARY / %TMP / %TEMP above. Though I am (slowly) coming round to the view that * Set X and %SET{"X"}%%EXPORT{"X"}% should be the same thing, and thus %SET would be correct.

-- CrawfordCurrie - 22 Apr 2012

Wouldn't it be totally confusing not to call it %SET ?

-- MichaelDaum - 23 Apr 2012

Sven, can you elaborate more your concerns and what we can do to address it?

-- MichaelDaum - 26 Apr 2012

I think I've decided that I need to see a grouping of unit tests - usage of SET in weird places like tmpl files, ADDTOZONE's etc, especially for values that are set in SitePrefs.

If you get to them before I do, great.

secondly, I'm still contemplating the different implications (and confusions) of *Local, *Set META:PREFERENCE and then adding a new one (or is that two?)
%SET{"CM" value="$percntDEFAULT{default=$quotebanana$quote}$percnt cm"}%

would mean %CM% would be 'banana' and %CM{"two"}% would be 'two'

-- SvenDowideit - 26 Apr 2012

Other things I'm liking is that we can do away with the 'copy these 12 settings to your SitePrefs ' for skin themes - we just make the topic a viewTemplate and provide the functionality / skin settings via an EXPORT

So this feature will allow us to convert skins into WikiApps, and begins to merge skin and app tmpl's

on the other hand, I'm not overjoyed about EXPORT - throwing more values into the GLOBAL namespace means we will get app developers clashing and un-debuggably over-writing other's values. (At least in the *Set and META:PREF case, it is possible to SEARCH / grep for where the value is set.

(as an eg, imagine several people build wikiapps that %EXPORT{"AGE, and then someone shows all of them in a dashboard page - the globally exported values will clash, and everyone has to needlessly re-write.)

Instead, I think we need to become more structured in how we intend to RETURN name-value pairs, and allow the caller to choose new names

As far as I can see, EXPORT=/=RETURN only has meaning for transclusion - INCLUDE/SEARCH/QUERY/**** do we have a canonical list of transclusion operators?,

%SET{"AGE" value="21"}%
%SET{"CM" value="$percntSET{$quotHEIGHT$quot value=$quot$percntDEFAULT$percnt$quot}$percnt$percntDEFAULT$percnt cm"}%

%STARTSECTION{"age_info"}%
At age %SET{"YOUNG" value="13"}%%YOUNG%, my height was %CM{"165"}%. At age %AGE% it is now %CM{"185"}%
%RETURN{"YOUNG"}%%RETURN{"HEIGHT"}%
%ENDSECTION{"age_info"}%

%INCLUDE{"%TOPIC" section="" import="HEIGHT,RETURNED_AGE=YOUNG"}%

   * %AGE% is 21
   * %RETURNED_AGE% is 13
   * %YOUNG% is undef
   * %HEIGHT% is 185

(as you can see import matches better with EXPORT - so smile )

  • additionally - I'd rather we implement CM as a SECTION to avoid the escaping hell
    • %DEFAULT% cm

So as you can see, one of my concerns is what your cartoon references. Another is that I'm not sure what happens to SET's and EXPORT's that happen in tmpl files and Template Topics.

-- SvenDowideit - 27 Apr 2012

You mean something like this?

%SET{"foo" section="bar"}%

%STARTSECTION{name="bar" type="section"}% a complex value being computed here %ENDSECTION{name="bar" type="section"}%

Could be done using HERE documents (never expedrimented with them / did we actually implement them?)

And yes, neither am I overexcited with %EXPORT. The alternative would be to flag scope as part of the %SET macro directly.

-- MichaelDaum - 27 Apr 2012

Michael - are you going to do some work on my concern wrt the global-ness, or will this wait until I do? (I'm distracted atm)

-- SvenDowideit - 10 May 2012

As an total_noob , really the original Set should be "Setenv" and the %SET would be ok.

Unfortunately legacy. So, now: what about the %SETVAR{"foo", value="bar"}%? and its counterpart %GETVAR{foo}%? And would be helful in the SpreadsheetPlugin make $SETVAR / $GETVAR too and marking $SET as deprecated in few years.. wink

-- JozefMojzis - 10 May 2012

@Jozef: %SETVAR and %GETVAR would clash with SetVariablePlugin

@Sven, let's chat about it as soon as you are available again.

-- MichaelDaum - 10 May 2012

oh sht... so, need something other, but, don't call it SET. Its confusing. Ideas: ASSIGN, LET

-- JozefMojzis - 10 May 2012

Try to answer this: "Why the f* isn't it called the same?"

There definitely is a relation to the * Set way of setting variables / constants: this is storing values into the same namespace with variable names competing with each other in terms of priority.

That's why I think that we should have very very good reasons normal users understand not to call it %SET.

As I already said above already: anything else would rather be more confusing than helping for users lining out the differences between * Set and %SET, that definitely exist.

-- MichaelDaum - 11 May 2012

Set and Local - and both might be useful - SET being the =LOCAL=&Export version.

Michael - what I need is unit tests that show how the current implementation idea reacts in the plausibly difficult places - tmpl files, and weirdly nested situations - and how they then interact with FINALISED / ACL things.

The worst thing is that your cartoon is exactly what i'm concerned about. using %SET (with export) enables users to randomly change Global settings to random things - whereas at the moment, preferences in a context are immutable.

-- SvenDowideit - 02 Jul 2012

Finalize is a good point. It actually might be a hint that Foswiki::Func::setPreferencesValue shouldn't be allowed to change a finalized value on perl level either, and as a consequence nor would %SET being a pure interface to Foswiki::Func::setPreferencesValue.

So here's the question: Should finalized values be immutable even on perl level?

Sven, which weirdness do you expect in %TMPL + %SET mixups? I can't think of any abnormal case. If you have a good idea of what this might be, I'd be able to carve that into a unit test.

Changing global values is already possible using the other means of adding stuff to the prefs stack, e.g. in parametrized %INCLUDEs., isnt it. Everything happening inside that %INCLUDE is in danger of suffering from strange globals.

-- MichaelDaum - 02 Jul 2012

I didn't say I expected weirdness, I expect unit tests that show what happens in places were we are not (yet) expecting SET to be used.

no, parametrised INCLUDEs only flow on to SECTIONs that are called from within the preceeding INCLUDE.

this SET proposal proposes to do something more complicated that we have before - to allow adding and changing preferences values in the middle of a section

And then it talks about an EXPORT macro that changes the preference tree in a completely new way, in that it allows affecting the calling section, again, in the middle of the rendering of that section.

the core thus far has shied away from this kind of thing - we've been treating preferences as immutable within their context.

so the long & short is, write unit tests, see what the code does before and after the proposed change - I might be concerned about nothing, but the unit tests need to be written for the feature either way.

-- SvenDowideit - 03 Jul 2012

Okay, no problem. The unit tests will be pretty straight forward juggling a few constellations of reading and writing variables using %SET. Behavior will be guided by standard evaluation orders.

I won't implement an EXPORT macro. That's the idea of somebody else, not me, can't even remember who proposed that. So it will require a feature request of its own.

What I will implement is given in the #Implementation section above, plus some unit tests hammering it. See the #Tests above for some.

-- MichaelDaum - 03 Jul 2012

Sven, what else can I do to address your concerns?

-- MichaelDaum - 04 Jul 2012

So what I need, is to see how the adhoc evaluation changes the precedence of, and how non-topic evaluations can affect preferences.

You've not updated the docco to reflect what you're going to do, and how this will change preferences from 'evaluated before rendering' (essentially when a topic is loaded), and what happens when %SET is used in a template - both tmpl and topic Template.

I also wonder what fun things will be possible using SET in a topic creation or form definition topic

My concern is that the proposal does not cover these complicated things which may well require changing the proposal.

-- SvenDowideit - 09 Jul 2012

What is a non-topic evaluation? Whatever it is, why is it important? All %SET does is call Foswiki::Func::setPreferencesValue(). These values are set during parsing and thus take precedence over values set anywhere else. That's basically all.

Sorry, but I don't get what you mean, Sven.

The only issue I see is expressed by my #finalizeQuestion above that nobody seems to have an opinion on.

-- MichaelDaum - 09 Jul 2012

Maybe that's because immutual is somewhat difficult to understand. wink Does it mean "not mutual" or "distinct"? In any case I'm not nearly qualified enough to have an opinion on this matter. From the users point of view I'd like to be able to set/get variable values differently in applications via macro calls.

-- FranzJosefGigler - 09 Jul 2012

Basically, "finalize" means the variable can't be changed anymore. As things are right now, plugins still can change values of finalized preferences under the hood using a standard call to Foswiki::Func::setPreferencesValue() ... which %SET would be a frontend to.

There are two options to resolve this issue:

  1. fix Foswiki::Func::setPreferencesValue() not to be able to change an already finalized preference variable
  2. let %SET do the necessary things to respect finalization. this might duplicate the same logic done elswhere for plain * Set.

I'd personally strongly opt for (1). It shows that there actually is a deeper problem in the concept of finalization as the logic doesn't seem to be implemented at a point in the code path where it belongs to.

-- MichaelDaum - 09 Jul 2012

I removed the %EXPORT macro from the documentation. I was never intending to implement it and the person who added it is not committed to implement it, as far as I see.

To clarify: this proposal is only about implementing %SET as provided in the patch above. Any further extension to this needs a feature proposal of its own, and a different CommittedDeveloper, i.e. not me.

Note also that this rather simple and pretty much self contained feature proposal (implementation and tests provided) will soon become 3 (three) years old constantly being vetoed against it for unclear reasons, at least to me. I have steam to leave it in discussion mode for say another month before I will step back from this initiative and park this proposal.

-- MichaelDaum - 22 Jul 2012

12 days left before I wil step back from this work and park the proposal. Sven, you are still blocking this even though I addressed all of your concerns as far as I see.

-- MichaelDaum - 10 Aug 2012

Michael, why not flip this to CommunityVote? And above the word immutual is used. I suspect it meant to be immutable?

-- GeorgeClark - 10 Aug 2012

The CommunityVote has been set up.

-- MichaelDaum - 10 Aug 2012

How do we vote?

-- RuiProcopio - 10 Aug 2012

@RuiProcopio: Add your vote in a new line into the voting table above. -- MichaelLorenzen - 10 Aug 2012

My concern is basically that the effects of using SET in places like tmpl has not been addressed. I asked for unit tests to show what the currently proposed implementation would do so we could discuss those facts, but you have not done so.

Clearly, no one else cares about this.

So vote away.

-- SvenDowideit - 11 Aug 2012

I do not envision the security implications. But my gut feeling is that we will only find them by using this feature "in the real world", beta testing is not sufficient for this. Maybe the best approach would be to implement this feature with 2 safegards: - detailed logging to trace how it is used - a simple to activate "kill switch" to disable it in case a hole is discovered. (temporarily until a proper fix is designed)

-- ColasNahaboo - 11 Aug 2012

Hey. New guy here. As a Software Engineer, I fully understand the concepts of scope and order of operations. For those who have not read http://shop.oreilly.com/product/9781593274245.do I can see how those concepts can be confusing.

As a Board member of several mostly-volunteer non-profits, and sometimes Manager at work, I would really like to see a separation between the conversation about "will this confuse the user" and "is it a good idea". They are very different conversations that need very different processes. "Will this confuse the user" mostly about finding a differentiating name, like "setLocal" or "setInScope", but it's not a technology problem. You know in your heart that someone will come up with the right word and description.

"Is it a good idea" and "how should it be implemented so it doesn't break stuff" are technology questions which, as SvenDowideit said, is best proposed, tested, and discussed over a nice hot bowl of unit tests, where hypotheses on how the sky will fall can be factually disproved.

I feel mixing the two conversations is making consensus harder. I'm hoping here to help the decision process.

-- DavidKramer - 12 Aug 2012

Having said the above, I will squirt just a bit of lighter fluid on the fire wink How will this work differently than $percentCALC{"$SET(my_total, $SUM($ABOVE()))"}$percent from SpreadSheetPlugin? On the one hand, if it does the same thing, maybe this new feature is not needed, because SpreadSheetPlugin can be used. On the other hand, maybe it can be used to test those aforementioned hypotheses with no coding at all.

-- DavidKramer - 12 Aug 2012

AFAIK are macros set by SpreadSheetPlugin only available to SpreadSheetPlugin via %CALC("$GET ...}% and therefore not as 'valuable' as the here proposed %SET{ ... }%.

-- AndreLichtsteiner - 13 Aug 2012

I would propose some tests along the following idea:

  1. %SET("FINALPREFERENCES")%or %SET("DENYTOPICVIEW")% in a template
  2. add the template to the skin by ?cover=myskin
  3. see whether Foswiki honours all ACL and final preferences as it supposed to.

Along above test, I would be interested, what would be the sequence, in which the Macros get set? Topic before Template?

-- AndreLichtsteiner - 13 Aug 2012

(1) Yes, it should not be possible to %SET some of these special variables.

(3) %SET must not allow to alter variables listed in FINALPREFERENCES. Thats probably a flaw of the underlying perl API already as has been noticed in the above discussion.

I am not sure in how far (2) leads to any test of a %SET. Templates are expanded well before anything else with none of the macros being expanded at all, including %SET. So variables can't affect template expansion that way.

-- MichaelDaum - 17 Aug 2012
 
Topic revision: r82 - 05 Jul 2015, GeorgeClark
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy