Item14007: weird malfunction in INCLUDE

I have some code which I want to INCLUDE from a file over in pub. Here's a cut-down version that exhibits the problem:

// foo!

  for (int i=0; i<lim_cycles; i++) {
  }

//SYSIN DD DUMMY

As shown above, this code renders properly when it's located in the topic page. However, it does not render properly when I bring it in from another file. Specifically, all of the line breaks following the "for" line go away, like this:

// foo!

  for (int i=0; i<lim_cycles; i++) { } //SYSIN DD DUMMY

The INCLUDE line I was using when the bug surfaced was something like this:

%INCLUDE{"https://foswiki.org/pub/.../foo.txt"}%

I can work around the problem by changing the line to something like this:

%INCLUDE{"https://foswiki.org/pub/.../foo.txt" raw="on"}%

Live example

%INCLUDE{"%ATTACHURL%/foo.txt"}%

// foo!

  for (int i=0; i<lim_cycles; i++) {   } //SYSIN DD DUMMY 

-- RichMorin - 05 Mar 2016

When you include a URL, it's treated as an HTML page, and renders per the normal HTML rendering. Whitespace on adjacent lines is never preserved in HTML unless it's part of a <pre> block. But in this case that won't work either because your text file contains unescaped < as well. I think you've found the right solution.

-- GeorgeClark - 05 Mar 2016

I think it's the unescaped < in the text. the INCLUDE of URLs' is documented as: :

1. ) Alter relative links to point back to originating host,
2. ) Remove some basic HTML tags (html, head, body, script) and finally
3. ) Remove newlines from HTML tags spanning multiple lines. (This compression step rewrites unmatched <'s into < entities unless bypassed)

I suspect that it's this last option. The unmatched < is subject to tag compression first, removing the blank lines, and then converted to the entity.

Confirmed, yes, it's the tag compression. Processing HTML using regular expressions can be difficult. I'll confirm the task, but it needs a developer interested in rewriting the parser in Foswiki::IncludeHandlers::http.

-- GeorgeClark - 05 Mar 2016

"the unescaped < in the text"

Inside a verbatim, inside an INCLUDE, a < is used?

That feels like a bug to me. There's a verbatim in there.

-- VickiBrown - 05 Mar 2016

I already confirmed the task as a bug. But remember, INCLUDE of a http:// web page is including HTML not topic markup. The include code doesn't look for any TML / pseudo-HTML, like verbatim tags. It just happens to go through our rendering on the way to a topic. But INCLUDE process HTML received via HTTP get. verbatim is not HTML.

-- GeorgeClark - 06 Mar 2016

Test with at textarea:

// foo! for (int i=0; i<lim_cycles; i++) { } //SYSIN DD DUMMY

Seems fine.

Test loading it into a skin=text section:

test1

%STARTSECTION{"test1"}%%INCLUDE{"%ATTACHURL%/foo2.txt"}%%ENDSECTION{"test1"}%

Now wrapping it into <literal>:

test2

%STARTSECTION{"test2"}%<literal>%INCLUDE{"%ATTACHURL%/foo2.txt"}%</literal>%ENDSECTION{"test2"}%

Seems fine.

But testing with an a <liteal> and/or a <pre> wrapper directly into the html page does not work. But probably for different issues in Foswiki::Render, not necessarily in %INCLUDE

This should be fine as well

<iframe src="%ATTACHURL%/foo2.txt"></iframe>

Long story short: you can't mix different content types, i.e. HTML and plain text as is. Either separate both (iframe, separate window) or convert plain text to HTML first and then inject it into the rest of the HTML page. The example text is especially problematic as it is programming code containing a < sign which is interpreted incorrectly when the content type does not match. For similar reason spurious <p> elements are injected when interpreting the code as TML.

Basically you need an include that rewrites content the way <verbatim> does.

%INCLUDE{... verbatim="on"}%

-- MichaelDaum - 06 Mar 2016

Thanks for the explanations! FYI, I'm using this hack to combine commentary, program text, and a Zip archive on a single wiki page (e.g., http://wiki.cfcl.com/Projects/Access/Utiles/NB/AS). I edit and test the code on machine foo, using the Arduino IDE. I then run a script (on foo) which creates the *.txt and *.zip files and scp's them into a single folder in the pub area on machine bar.

-- RichMorin - 06 Mar 2016

Commenting is disabled while not logged in