Item1358: EditTablePlugin deep recurses in many cases

pencil
Priority: Urgent
Current State: Closed
Released In: 1.0.5
Target Release: patch
Applies To: Extension
Component: EditTablePlugin
Branches:
Reported By: KennethLavrsen
Waiting For:
Last Change By: KennethLavrsen
EditTablePlugin can deep recurse.

I have some software people that have found out that new lines inside verbatim blocks are not seen as new lines by the table parser.

Not even I knew that. They use it to create ASCII tables inside a table cell.

A table can look like this (real text replaced by garbage).

Scenario
+--------------------------+-----+-----+-----+-----+-----+
|                          | xx1 | xx2 | xx3 | xx4 | xx5 |
+--------------------------+-----+-----+-----+-----+-----+
| fsdfdsaf                 |  4  |  4  |  4  |  3  |  3  |
| fdfasdasfasdfasdffdsafa  | 11  |  8  | 13  | 12  |  7  |
+--------------------------+-----+-----+-----+-----+-----+

and then we make another table the same way.

Scenario
+-------------------------------+
|   xx    |1 2 3 4 5 6 ... 15 16|
+---------+---------------------+
|fdsffsdf |0 0 1 1 1 2 ...  1  1|
|fdsfdsfs |0 1 0 1 2 2 ...  0  1|
+---------+---------------------+

In 1.0.4 this causes a deep recursion.

The condition for the recursion is

TWO or more table must use this syntax The verbatim start and end must be on seperate lines.

-- KennethLavrsen - 23 Mar 2009

The verbatim part of the equation is not the root cause of the deep recursing.

It will be nice if the verbatim or pre can still work after a fix - but the deep recurse is an issue and after I now know more it is of a nature I have to elevate to urgent.

This simple test case also produces the recurse (verbatim tags not part of the example)

%EDITTABLE{ format="| text, 14 |"  }%
| x
y |
 

%EDITTABLE{ format="| text, 15 |"  }%
| x 
 x |

So the root cause is related to this sequence

  • First an EDITTABLE tag
  • Then the beginning of a table line that is not finished
  • Then the rest of the table line
  • Then a new EDITTABLE tag
  • New beginning of table line that is not finished
  • Then the rest of the table line

It seems the unfinished lines is the what the plugin cannot cope with.

In the case of no verbatim - the plugin naturally cannot see the next lines as anything sensible and should just ignore it. But it must never deep recurse. Such behavior is potential DOS attack vector.

-- KennethLavrsen - 24 Mar 2009

I have started debugging and as I have to leave for work my last conclusion is the loop happens in commonTagsHandler during core::processText in line 567 where the code calls

$resultText = Foswiki::Func::expandCommonVariables($resultText)
          if ( !$doEdit && ( $mode & $MODE->{READ} ) );

Adding a poor mans debug on the line before printing out the $resultText I get this repetitive behaviour. Test topic is the simple example above.

[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <a name="edittable1"></a>, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <div class="editTable">, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <form name="edittable1" action="http://merlin.lavrsen.dk/foswiki10/bin/viewauth/Myweb/TableRecursionProblem#edittable1" method="post">, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="ettablenr" value="1" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="etedit" value="on" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] | x, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] y |, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11]  , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <!--%EDITTABLESTUB{2}%-->, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] | x , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11]  x |, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="etrows" value="0" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input class="editTableEditImageButton" type="image" src="%PUBURL%/%SYSTEMWEB%/EditTablePlugin/edittable.gif" alt="%MAKETEXT{"Edit this table"}%" /> </form>, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] </div><!-- /editTable -->, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <a name="edittable1"></a>, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <div class="editTable">, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <form name="edittable1" action="http://merlin.lavrsen.dk/foswiki10/bin/viewauth/Myweb/TableRecursionProblem#edittable1" method="post">, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="ettablenr" value="1" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="etedit" value="on" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] | x, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] y |, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11]  , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <!--%EDITTABLESTUB{2}%-->, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] | x , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11]  x |, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="etrows" value="0" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input class="editTableEditImageButton" type="image" src="%PUBURL%/%SYSTEMWEB%/EditTablePlugin/edittable.gif" alt="%MAKETEXT{"Edit this table"}%" /> </form>, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] </div><!-- /editTable -->, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <a name="edittable1"></a>, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <div class="editTable">, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <form name="edittable1" action="http://merlin.lavrsen.dk/foswiki10/bin/viewauth/Myweb/TableRecursionProblem#edittable1" method="post">, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="ettablenr" value="1" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="etedit" value="on" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] | x, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] y |, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11]  , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <!--%EDITTABLESTUB{2}%-->, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] | x , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11]  x |, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input type="hidden" name="etrows" value="0" />, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <input class="editTableEditImageButton" type="image" src="%PUBURL%/%SYSTEMWEB%/EditTablePlugin/edittable.gif" alt="%MAKETEXT{"Edit this table"}%" /> </form>, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] </div><!-- /editTable -->, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] , referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <a name="edittable1"></a>, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <div class="editTable">, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome
[Tue Mar 24 09:38:19 2009] [error] [client 192.168.1.11] <form name="edittable1" action="http://merlin.lavrsen.dk/foswiki10/bin/viewauth/Myweb/TableRecursionProblem#edittable1" method="post">, referer: http://merlin.lavrsen.dk/foswiki10/bin/view/Myweb/WebHome 

-- KennethLavrsen - 24 Mar 2009

I have additional reports now from my users and this is becomming a real problem

If it is a problem for me it is a problem for many other installations.

I wonder why noone steps in and tries to at least help with the debugging frown, sad smile

The new cases I have seen are more serious than the above.

If you have a table and someone by accident have a line after the table with white space followed by a vertical bar | and followed by more EDITTABLE tags you also get same deep recursion. It seems the parser cannot find the end of the table properly. Anything out of the ordinary makes the code loop. When this is fixed I will need to release a 1.0.5.

-- KennethLavrsen - 26 Mar 2009

I have done a lot of debugging now.

Here is a status.

As far as I can see - everything that happens in the beforeCommonTagsHandler is correct.

And in the commonTagsHandler we are doing the right thing in Core::initIncludedTopic and Core::Core::parseTables

And when we come to Core::process things go wrong.

When I step through the Core::processText sub thing look OK until we reach handleSearchResultsBelowEditTables( $topicText, $paramTableNr, $tableData );

If I disable this line and this way disable this feature that appends search result in the form of a table, the plugin behaves correctly.

So there is something goofy with handleSearchResultsBelowEditTables

I continue debugging.

-- KennethLavrsen - 26 Mar 2009

OK I have the bug nailed.

It is this regex which is not correctly identifying the result of a table search.

s/(\s+)((\s*)\|.*\|)\s+/addSearchResultsTableTextToTableObject($_[2], $_[1], $1, $2, $3)/geos;

Next step is fixing this regex.

-- KennethLavrsen - 26 Mar 2009

I am currently testing this regex from hell.

s/([ ]*\n)((([ \t]*\|[^\n]*\|)[ \t]*\n)+)/addSearchResultsTableTextToTableObject($_[2], $_[1], $1, $2, $3)/geos;

So far it works.

-- KennethLavrsen - 26 Mar 2009

So here is the final regex from hell. But it works.

s/([ ]*\n)((?:[ \t]*\|[^\n]*\|[ ]*\n)+)/addSearchResultsTableTextToTableObject($_[2], $_[1], $1, $2, $3)/geos;

Arthur should have helped with a new unit test to prevent the problem as I write this.

-- KennethLavrsen - 26 Mar 2009

ItemTemplate edit

Summary EditTablePlugin deep recurses in many cases
ReportedBy KennethLavrsen
Codebase 1.0.4
SVN Range Foswiki-1.0.0, Thu, 08 Jan 2009, build 1878
AppliesTo Extension
Component EditTablePlugin
Priority Urgent
CurrentState Closed
WaitingFor
Checkins distro:b3e56de250fe distro:96d3193a3a3c distro:422571c9bdbd distro:d20e2a78a5f2
TargetRelease patch
ReleasedIn 1.0.5
Topic revision: r12 - 25 Apr 2009, KennethLavrsen
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy