Item1736: Macro expansion of %TOPIC% (or %BASETOPIC% ) in formatted search doesn't point to original topic

Description

We have a DataForm filled in a way, that one of the fields is dynamically filled, using the %TOPIC% macro, e.g. Name : %TOPIC%. A formatted search with output of $formfield(Name) will not return the name of the original topic; instead, the %TOPIC% macro will get expanded to the name of the topic where the actual search took place.

I can't think of any scenario where this behaviour might be wanted. But even if so, there should be a way to workaround this. Using %BASETOPIC% instead of %TOPIC% doesn't have any effect, since the search results don't actually get INCLUDEd, but anything like that is necessary then.

---

I did some further testing on this issue:

• using the %SEARCH{}% parameter expandvariables="on" does not help, although I would expect it, since it "expands embedded macros before applying a FormattedSearch on a search hit" (see VarSEARCH)
• the %FORMFIELD{}% macro shows the same behaviour
• same for ImageGalleryPlugin, e.g. if the comment of an image file contains %TOPIC% and the comment is shown in title or thumbtitle (see Item1813 "BTW:...")

So this problem is quite widespread. Unless I miss the proper workaround for this (and I really hope so...), I really think, this is a quite urgent problem, since there must be a way to expand macros like %TOPIC% before applying %SEARCH{}%, %FORMFIELD{}% and functions alike.

How about extending the functionality of the %BASETOPIC% macro to be recognized by these functions as well?

Setting priority to urgent, since nobody answered yet...

-- PhilippLeufke - 08 Jul 2009

Im not sure, if I got your problem correct.

If you literally put a %TOPIC% into one of the formfields, it is saved unexpanded as %TOPIC%. (You can see, what is saved, if you add a "?raw=all" url parameter to the view url).

Logically SEARCH or any other method accessing the formfield does not get the "old" expanded value, but either the literal %TOPIC% or the locally expanded version of it.

Why do you put the literal %TOPIC% into the formfield if you actually want the expanded value in there?

-- OliverKrueger - 08 Jul 2009

Short answer to this question: because it is not static.

OK, maybe I boiled the problem down too much; in fact it is more complicated. The approach is heading towards a more relational database, so one formfield is filled by processing information of other form(s).

This is the original formfield content:

%SEARCH{
"BatterySamplesForm.SampleID='%FORMFIELD{"SampleMaterials" topic="%TOPIC%"}%'"
type="query" exclude="*Template" nonoise="on" limit="1"
format="$formfield(ObjectiveMaterial)" }%

So locally,i.e. when viewing the topic where the form is attached, this gets expanded like expected, but when extracting this formfield from a %SEARCH{}% or with %FORMFIELD%, it is broken.

I understand what you say about when the macros get expanded, although I feel that expandvariables="on" should move the expansion to the "old" topic entirely. In the case like it is, we need something similar to %BASETOPIC% which allows to refer to original topics. Right?

-- PhilippLeufke - 09 Jul 2009

I see in the comments above that this problem is more widespread than just the %FORMFIELD{}% macro, but is an issue with SEARCH as well. Reflecting on this reality, it will be important to resolve macros when the topic is saved so that resolution of macros is not required during searches. Therefore, it may be necessary to maintain both the raw formfield content as well as the resolved content, thereby allowing fast searches but also facilitate use of simple macros in formfields, such as %TOPIC% %ATTACHURLPATH%, etc.

One solution is to add a formfield attribute in the DataForm topic that indicates the formfield to be resolved upon topic save and the raw content to be saved as an additional field in the META HTML markup. When in edit mode, the raw content would be displayed and edited. When saved, the formfield is then resolved to constant data that can be easily searched, cached, etc. Without the formfield attribute, the formfield would be processed as it is today, and overhead of attempting to resolve macros in any formfields would be skipped.

This should fix the problem both for use of the FORMFIELD macro (when another topic is referenced) and SEARCH when formfields are referenced. However, it suffers from the drawback that formfield content is not truly dynamic, and is fixed upon each save of the topic.

-- RaymondLutz - 13 Jul 2009

Thanks! Now I understand the performance issues (plain text search vs. macro expansion). Looks like the application we wanted to build is pushing foswiki's plain text database concept to the limit (nevertheless I like the concept...).

Any other idea how to workaround the problem?

BTW: But what I still don't understand is the expandvariables="on" option. In which cases does it work and why doesn't it work here?

-- PhilippLeufke - 13 Jul 2009

Apparently, data in formfields is intended to be mostly static in nature, so that it can (hopefully) be moved to a pure database (like mysql) for improved performance. In that case, the body of the topic would need to be maintained in a "blob", I would think, as a field in the database.

What we don't want is a scheme where we must resolve the macros in the formfields whenever we do a search. That will result in very poor performance and will not leverage any database search queries.

Instead, as I have suggested, we would need to pre-resolve the contents of the form so that it will not require any modification during the search. I want to additionally point out that allowing the %FORMFIELD{}% macro within a formfield that refers to a formfield in the same topic is a prescription for disaster, and preferably, any macros used in formfields would be simple (like definitions, i.e. %TOPIC%, %ATTACHURLPATH%, etc.).

One workaround that I use is that you can create a relational database structure by initializing the value when the topic is created. For example, we maintain a member (customer) database and can add a transaction for each one, and link it back to the customer record. Here is the "Add transaction" link:

New Transaction

in this case, it is defined for use in the TransactionHeader. (I always use headers in automatically generated topics because they are easy to change and then all topics get the same thing.) In any case, when the new topic is generated, it has the correct parent topic initialized in the form (MembTopic). This workaround won't work for attached images, for example, because they are added later.

-- RaymondLutz - 13 Jul 2009

I think the discussion has covered the main points, and this is a case of unfulfilled expectation, rather than a bug.

No action.

-- CrawfordCurrie - 15 Dec 2009