MongoDBPlugin Development

This is the topic to discuss development of download MongoDBPlugin

help If you need support, go to Support.MongoDBPlugin where you can ask questions and find answers to previously asked questions. warning If you want to report a bug, or a feature request, go to Tasks.MongoDBPlugin where you can see already submitted issues and where you can submit a new bug report or feature request.

Active Items

Id Summary Priority Current State Creation Date Last Edit
Item9950 MongoDB unit test failures due to sort ordering Normal New 04 Nov 2010 - 04:32 04 Nov 2010 - 04:32
Item10463 WIBNIF we had a distinct() operator Enhancement New 10 Mar 2011 - 00:26 10 Mar 2011 - 00:26
Item10628 MongoDB query does not do numeric OP's on number fields / prefs correctly, because atm, everything isa string. Urgent Confirmed 12 Apr 2011 - 04:12 12 Apr 2011 - 04:12
Item10610 Foo~'*Blah' is massively slower than Foo='Blah' Low New 08 Apr 2011 - 03:33 13 Apr 2011 - 01:43
Item10532 WIBNIF we allowed queries to go to slaves (slave_ok) Enhancement Waiting for Feedback 24 Mar 2011 - 05:21 20 Apr 2011 - 10:35
Item10702 fix the non-aliased version of Query's for META: in MongoDBPlugin Normal New 04 May 2011 - 01:33 04 May 2011 - 01:33
Item10653 The simpler WebTopicList is slower than WebIndex Low New 19 Apr 2011 - 03:23 13 May 2011 - 05:08
Item10652 MongoDBPlugin milestone 5 Normal New 19 Apr 2011 - 03:22 17 May 2011 - 03:28
Item10704 Minimise chattiness (with Store) of paged SEARCHes Normal New 04 May 2011 - 06:11 18 May 2011 - 02:34
Item10748 foswiki Response::body() corrupting international chars if the underlying store is utf-8 and Site-CharSet is set to utf-8 Urgent Waiting for Feedback 16 May 2011 - 07:57 21 May 2011 - 14:56
Item10761 make MongoDBPlugin not crash foswiki if MongoDB service is not running Low New 18 May 2011 - 03:16 23 May 2011 - 10:34
Item10783 MongoDB hoisting generates where clauses after simplification Normal New 24 May 2011 - 05:55 24 May 2011 - 05:55
Item10829 WIBNIF we didn't have to store this redundant _raw_text stuff Enhancement New 02 Jun 2011 - 06:16 02 Jun 2011 - 06:16
Item10947 I think mongodb has broken the WikiGroups view Urgent Waiting for Feedback 07 Jul 2011 - 07:23 07 Jul 2011 - 07:23
Item11022 Enhance order param so we can sort on any VarQUERY/"getField" expression Enhancement New 05 Aug 2011 - 05:16 05 Aug 2011 - 05:16
Item11040 MongoDB error in regex search and match query Normal New 12 Aug 2011 - 20:46 12 Aug 2011 - 20:46
Item10740 MongoDBPlugin breaks Foswiki unless all replicas are up Normal New 13 May 2011 - 05:19 16 Aug 2011 - 03:03
Item11061 lc(Field) =~ lc('something') failing again Normal New 24 Aug 2011 - 06:57 24 Aug 2011 - 06:57
Item11244 Item11215 introduced a filterByDate method, which MongoDBPlugin lacks Urgent New 08 Nov 2011 - 13:13 08 Nov 2011 - 13:13
Item11269 JQGridPlugin + MongoDB + large web + default sort = empty grid Normal Being Worked On 21 Nov 2011 - 00:32 21 Nov 2011 - 02:49
Item10328 use the writeDebug, luke! Enhancement New 08 Feb 2011 - 04:40 08 Dec 2011 - 09:00
Item11475 Mongo ignores form names when matching fields. Urgent New 25 Jan 2012 - 23:42 25 Jan 2012 - 23:42
Item11496 _updateTopic tries to set meta with undef name key Urgent Confirmed 03 Feb 2012 - 01:07 03 Feb 2012 - 01:22
Item11551 MongoDBPlugin still using the old loadedRev=0 semantics Normal Confirmed 23 Feb 2012 - 05:29 16 Mar 2012 - 07:33
Item11672 TOPICINFO or rcslite with mongo missing topics Urgent Waiting for Feedback 19 Mar 2012 - 10:29 21 Mar 2012 - 11:56
Item11811 MongoDB silently ignores topics which have no _latestIsLoaded version Normal Waiting for Feedback 03 May 2012 - 04:10 03 May 2012 - 04:10
Item11885 Ambiguous Foswiki ACLs confuse MongoDB ACL cache Enhancement New 22 May 2012 - 02:17 22 May 2012 - 02:17
Item12224 mongodb remove web should be recursive Urgent Confirmed 04 Nov 2012 - 12:08 07 Nov 2012 - 11:52
Item10670 JS Error: SyntaxError: missing ) in parenthetical nofile_a:0 Normal New 22 Apr 2011 - 23:17 12 Nov 2012 - 06:11
Item12222 autobuild with MongoDBPlugin Normal Being Worked On 03 Nov 2012 - 13:15 15 Nov 2012 - 03:44

Discussion

Preliminary reading

The official mongo documentation is quite good, but a little sparse for examples of setting up shards that consist of replica sets. Useful links (these have diagrams that are probably less confusing than those found here):

MongoDB Installation

Use the packages provided by 10gen. They release often; apparently too often for the major distros to keep up.

Some notes here moved to MongoDBNotes - see the old version

-- PaulHarvey - 14 Oct 2010

using the MongoDBPlugin version comited to svn today, on foswiki 1.1.0, on a dataset of 25,000 dataform using topics, and a query using the ~ operator.

  • 192.168.1.130 running mongoDB - ~2seconds
  • =192.168.1.103=topic store and query based - ~5.5seconds

both these computers are identical otherwise, 2GRam c2d 1.8, horridly slow disk... running apache 2.2 and using perl CGI only (yes, they both have mongodb installed..).

quad:/home/sven# ab "http://192.168.1.130/f/bin/view/Lauries/TestQuery1"
This is ApacheBench, Version 2.3 <$Revision: 1.18 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.130 (be patient).....done


Server Software:        Apache/2.2.16
Server Hostname:        192.168.1.130
Server Port:            80

Document Path:          /f/bin/view/Lauries/TestQuery1
Document Length:        37370 bytes

Concurrency Level:      1
Time taken for tests:   1.954 seconds
Complete requests:      1
Failed requests:        0
Write errors:           0
Total transferred:      37738 bytes
HTML transferred:       37370 bytes
Requests per second:    0.51 [#/sec] (mean)
Time per request:       1954.077 [ms] (mean)
Time per request:       1954.077 [ms] (mean, across all concurrent requests)
Transfer rate:          18.86 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        6    6   0.0      6       6
Processing:  1949 1949   0.0   1949    1949
Waiting:     1939 1939   0.0   1939    1939
Total:       1954 1954   0.0   1954    1954
quad:/home/sven# ab "http://192.168.1.103/f/bin/view/Lauries/TestQuery1"
This is ApacheBench, Version 2.3 <$Revision: 1.18 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.103 (be patient).....done


Server Software:        Apache/2.2.16
Server Hostname:        192.168.1.103
Server Port:            80

Document Path:          /f/bin/view/Lauries/TestQuery1
Document Length:        37369 bytes

Concurrency Level:      1
Time taken for tests:   5.409 seconds
Complete requests:      1
Failed requests:        0
Write errors:           0
Total transferred:      37737 bytes
HTML transferred:       37369 bytes
Requests per second:    0.18 [#/sec] (mean)
Time per request:       5409.196 [ms] (mean)
Time per request:       5409.196 [ms] (mean, across all concurrent requests)
Transfer rate:          6.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        5    5   0.0      5       5
Processing:  5404 5404   0.0   5404    5404
Waiting:     5393 5393   0.0   5393    5393
Total:       5409 5409   0.0   5409    5409


milestone 1

Latest progress - DebugLogPlugin (in svn) adds a http header item telling you how long it took to render the topic request X-Foswiki-Monitor-Debuglogplugin-Rendertime (sans xfer time), and MongoDBPlugin adds a list of times spent querying mongo X-Foswiki-Monitor-Mongodbplugin-Lastquerytime

note that these number differ from the above due to removal of network latencies, and the timer starts after perl compilation.

These number are again for a raw text regex on the raw topic text, which is not a good use of mongodb's capabilitys

sven@quad7:~$ curl -I http://hp4/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Wed, 05 Jan 2011 03:14:44 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: noQuery
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 3.51224 wallclock secs ( 2.96 usr  0.15 sys +  0.10 cusr  0.41 csys =  3.62 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=6c02712d9f41f3a13dbd7572fcae3464; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

sven@quad7:~$ curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Wed, 05 Jan 2011 02:59:58 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.410422
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 1.58916 wallclock secs ( 0.84 usr +  0.08 sys =  0.92 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=40dabda1b3a4c136e0d0077d67ea77f7; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

milestone 2

(from 2 client systems at the same time, so perl is hit a bit harder)
sven@hp2:~$ curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
sven@hp1:~$ curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Tue, 08 Feb 2011 04:49:02 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.134943
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 1.64855 wallclock secs ( 0.95 usr +  0.10 sys =  1.05 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=09718e0fc9e873c6488c96ee658e9c5f; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

or from just one client (more comparible with the previous results)
sven@hp1:~$ curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Tue, 08 Feb 2011 04:50:38 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.131832
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 1.59396 wallclock secs ( 0.92 usr +  0.11 sys =  1.03 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=32d89c2da15b2d9abb30b895481d0620; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-

Milestone 3

day 1

hacked code
sven@hp1:~$ curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Fri, 11 Feb 2011 01:13:23 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.000162
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 1.48638 wallclock secs ( 0.88 usr +  0.09 sys =  0.97 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=62e5b4873a3630905205675be4e0af83; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1
somewhat puzzling - All I did was to use the new trunk listener API to load the data from mongoDB (one at a time, slow and nasty) rather than from .txt file.

much more work is needed, as clearly the roudtrips are silly, and we're still hitting the disk .txt files for topicExists etc, due to legacy arch. but it suggests that its worth doing.
then with slight changes and comited as Foswikirev:10679
sven@hp1:~$ curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Fri, 11 Feb 2011 04:00:54 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.000163
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 1.48613 wallclock secs ( 0.88 usr +  0.09 sys =  0.97 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=e9a053b93a3aad505b18c4cb0e60018b; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

-- SvenDowideit - 11 Feb 2011

after another week of fixing bugs in the MetaCache - Foswikirev:10783


sven@hp1:~$ time curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Wed, 23 Feb 2011 12:07:05 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.000109
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 1.33302 wallclock secs ( 0.65 usr  0.11 sys +  0.13 cusr  0.03 csys =  0.92 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=66a5b7d7803435aed2a036f5046ddb8a; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1

-- SvenDowideit - 23 Feb 2011

end of milestone 3

hp1 is currently down, as my main server's motherboard is dead frown

so the tests come from the same HW - with the server's disks in it.

sven@quad:~$ time curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Mon, 28 Mar 2011 07:59:45 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.570112
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 1.30385 wallclock secs ( 0.61 usr +  0.05 sys =  0.66 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=decc6e6d36c3eb32a8ad701ebbbed350; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1


real   0m1.867s
user   0m0.004s
sys   0m0.004s

  • end of m2 == 1.59396
  • end of m3 == 1.30385

for a gain of just under 300ms.

much greater differences should exist for larger result sets, but we're working on that further for m4.

-- SvenDowideit - 28 Mar 2011

Milestone 4

day 1

changed the schema so that there is one database per web, allowing us a larger number of indexes per web, and possibly allowing mongodb to manage memory more interestingly.

need to totally re-import the webs..

sven@quad7:~$ time curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Tue, 05 Apr 2011 04:13:29 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.162753
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 1.20481 wallclock secs ( 0.58 usr +  0.06 sys =  0.64 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=724e771ebd5dd0460ddc2ee36c153194; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1


real   0m1.761s
user   0m0.000s
sys   0m0.000s

thats another 100ms faster - not at all what I expected.

-- SvenDowideit - 05 Apr 2011

I keep forgetting to do proper comparisons - over the months, I've been making foswiki core changes to fix performance issues that show themselves as bottlenecks in the MongoDBPlugin results, and so foswiki trunk is now faster too - see

sven@quad:~$ ab http://hp4/~sven/core/bin/view/Lauries/TestQuery1
This is ApacheBench, Version 2.3 <$Revision: 1.18 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking hp4 (be patient).....done


Server Software:        Apache/2.2.16
Server Hostname:        hp4
Server Port:            80

Document Path:          /~sven/core/bin/view/Lauries/TestQuery1
Document Length:        38502 bytes

Concurrency Level:      1
Time taken for tests:   3.800 seconds
Complete requests:      1
Failed requests:        0
Write errors:           0
Total transferred:      39006 bytes
HTML transferred:       38502 bytes
Requests per second:    0.26 [#/sec] (mean)
Time per request:       3800.435 [ms] (mean)
Time per request:       3800.435 [ms] (mean, across all concurrent requests)
Transfer rate:          10.02 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:  3800 3800   0.0   3800    3800
Waiting:     3793 3793   0.0   3793    3793
Total:       3800 3800   0.0   3800    3800
and
sven@quad:~$ time curl -I http://hp4/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Mon, 18 Apr 2011 05:11:54 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 3.61898 wallclock secs ( 2.94 usr  0.19 sys +  0.07 cusr  0.54 csys =  3.74 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=4f8c7da44075f8798fcbc59995188c6f; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1


real   0m4.038s
user   0m0.000s
sys   0m0.008s

so foswiki 1.2.0 should be 1.6 seconds faster than 1.1, whereas with MongoDBPlugin, you will get even more speed, and added scalability.

-- SvenDowideit - 18 Apr 2011

mmm, ok, and further comparisons, same query, once as cgi, and then the second using mod_fcgid


sven@quad7:~$ time curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Tue, 19 Apr 2011 04:52:57 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.075974
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 0.810351 wallclock secs ( 0.59 usr +  0.06 sys =  0.65 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=120d695a6eb40773c2a2da893f10e270; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1


real   0m1.365s
user   0m0.000s
sys   0m0.000s
sven@quad7:~$ time curl -I http://hp3/~sven/core/bin/foswiki.fcgi/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Tue, 19 Apr 2011 04:53:05 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/foswiki.fcgi/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.076134, 0.051091
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 18.7475 wallclock secs ( 1.11 usr  0.12 sys +  0.01 cusr  0.00 csys =  1.24 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=8805a10a8b700ff2aca180342bcf7759; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1


real   0m0.717s
user   0m0.000s
sys   0m0.000s

so far, that means we've taken and uncached query over a large web from ~5seconds to 0.7seconds on the same hardware, by offloading the querying to an indexed structured query service, and by avoiding the ~500ms startup cost of CGI

pleasing (m4 continuing, but mostly on non-speed related issues)

-- SvenDowideit - 19 Apr 2011

complete

m4 is almost complete - I've got one more index to add (that should not impact the Glossary queries at all), so here goes with the final-ish numbers:

note that in the last 2 weeks, we've added the _entire topic history into the database, so its now uses about 3 times as much memory as before - on this 32bit test server, it is almost maxing out the 2.5G MongoDb limit, and the server starts to swap.


sven@quad:~$ time curl -I http://hp3/~sven/core/bin/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Wed, 04 May 2011 10:51:44 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.090679
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 0.983436 wallclock secs ( 0.61 usr +  0.05 sys =  0.66 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=e6337d30048658ddf3665ce0f031de77; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1


real   0m1.558s
user   0m0.000s
sys   0m0.008s
sven@quad:~$ time curl -I http://hp3/~sven/core/bin/foswiki.fcgi/view/Lauries/TestQuery1
HTTP/1.1 200 OK
Date: Wed, 04 May 2011 10:51:48 GMT
Server: Apache/2.2.16 (Debian)
X-Foswikiuri: /~sven/core/bin/foswiki.fcgi/view/Lauries/TestQuery1
X-Foswikiaction: view
X-Foswiki-Monitor-Mongodbplugin-Lastquerytime: 0.061654
X-Foswiki-Monitor-Debuglogplugin-Rendertime: 0.792923 wallclock secs ( 0.47 usr +  0.04 sys =  0.51 CPU)
Cache-Control: max-age=0
Set-Cookie: FOSWIKISID=ea8c2eccdd57a203952634bf62ac07fe; path=/; HttpOnly
Vary: Accept-Encoding
Content-Type: text/html; charset=iso-8859-1


real   0m0.890s
user   0m0.008s
sys   0m0.000s

so in the process of adding much more load on the server,its about 200ms slower than without. We might add an option to disable the topic histories, but thats low on our priorities - I have to migrate my test servers to 64bit frown which will change the results further.

-- SvenDowideit - 04 May 2011

Moved some admin notes to MongoDBNotes

-- PaulHarvey - 28 Sep 2011 using
<pharvey> +mongo 2.0.7, MongoDB 0.45, perl 5.14.2
<pharvey> +ACTION installing mongodb20-10gen_2.0.7_amd64.deb

Running unit tests
git submodule foreach 'git svn rebase ||:'
git submodule foreach 'git clean -fdx'
cd core && ./pseudo-install.pl -A developer MongoDBPlugin
sudo service mongodb stop
sudo rm -rf /var/lib/mongodb/*
sudo service mongodb start
visit bin/configure
set password
Extensions -> Enable MongoDB
Internationalization -> Locale -> {Site}{CharSet} = 'utf-8'
Store -> Impl -> Search & Query algos = MongoDB
save changes
Visit MongoDB's Extension Enabled warning, copy-paste into Store -> Impl -> Expert settings -> Listeners
Extensions -> MongoDBPlugin -> query_timeout = 20000
./rest /MongoDBPlugin/update -updateweb all -recurse on -revision on
/usr/bin/time -v perl -CSAD ../bin/TestRunner.pl FoswikiSuite.pm

-- PaulHarvey - 28 Sep 2011

-- PaulHarvey - 03 Nov 2012 - 07:06

lots of fixing gets svn 15907 down to 125 test failures

---++ Module Failure summary
FormDefTests has 1 unexpected results (of 10):
   * F: FormDefTests::test_Item6082
StoreTests has 2 unexpected results (of 38):
   * F: StoreTests::verify_CreateSimpleMetaTopic_StoreTests_RcsLite
   * F: StoreTests::verify_CreateSimpleMetaTopic_StoreTests_RcsWrap
SemiAutomaticTestCaseTests has 2 unexpected results (of 24):
   * F: SemiAutomaticTestCaseTests::test_TestCaseAutoFormattedSearch
   * F: SemiAutomaticTestCaseTests::test_TestCaseAutoSearchOrder
InitFormTests has 2 unexpected results (of 10):
   * F: InitFormTests::test_tmpl_form_new
   * F: InitFormTests::test_tmpl_form_new_params
VCStoreTests has 9 unexpected results (of 24):
   * F: VCStoreTests::verify_InconsistentTopic_getRevisionInfo_VCStoreTests_RcsLite
   * F: VCStoreTests::verify_InconsistentTopic_getRevisionInfo_VCStoreTests_RcsWrap
   * F: VCStoreTests::verify_Inconsistent_implicitSave_VCStoreTests_RcsLite
   * F: VCStoreTests::verify_Inconsistent_implicitSave_VCStoreTests_RcsWrap
   * F: VCStoreTests::verify_Inconsistent_saveAttachment_VCStoreTests_RcsLite
   * F: VCStoreTests::verify_Inconsistent_saveAttachment_VCStoreTests_RcsWrap
   * F: VCStoreTests::verify_NoHistory_implicitSave_VCStoreTests_RcsWrap
   * F: VCStoreTests::verify_NoHistory_saveAttachment_VCStoreTests_RcsLite
   * F: VCStoreTests::verify_NoHistory_saveAttachment_VCStoreTests_RcsWrap
QueryTests has 15 unexpected results (of 88):
   * F: QueryTests::test_match_lc_field
   * F: QueryTests::test_match_lc_field_simple
   * F: QueryTests::verify_array_dot_MongoDBQuery
   * F: QueryTests::verify_array_integer_index_MongoDBQuery
   * F: QueryTests::verify_atoms_BruteForceQuery
   * F: QueryTests::verify_atoms_MongoDBQuery
   * F: QueryTests::verify_boolean_bops_MongoDBQuery
   * F: QueryTests::verify_brackets_MongoDBQuery
   * F: QueryTests::verify_constants_MongoDBQuery
   * F: QueryTests::verify_form_name_context_MongoDBQuery
   * F: QueryTests::verify_numeric_bops_MongoDBQuery
   * F: QueryTests::verify_string_bops_with_mods_MongoDBQuery
   * F: QueryTests::verify_versions_on_cur_topic_MongoDBQuery
   * F: QueryTests::verify_versions_on_other_topic_BruteForceQuery
   * F: QueryTests::verify_versions_on_other_topic_MongoDBQuery
Fn_SEARCH has 28 unexpected results (of 524):
   * F: Fn_SEARCH::test_orderTopic
   * F: Fn_SEARCH::verify_attachmentSizeQuery2_ForkingSearch
   * F: Fn_SEARCH::verify_attachmentSizeQuery2_MongoDBQuery
   * F: Fn_SEARCH::verify_attachmentSizeQuery2_MongoDBSearch
   * F: Fn_SEARCH::verify_attachmentSizeQuery2_PurePerlSearch
   * F: Fn_SEARCH::verify_bad_order_BruteForceQuery
   * F: Fn_SEARCH::verify_bad_order_MongoDBQuery
   * F: Fn_SEARCH::verify_bad_order_MongoDBSearch
   * F: Fn_SEARCH::verify_date_param_ForkingSearch
   * F: Fn_SEARCH::verify_date_param_MongoDBQuery
   * F: Fn_SEARCH::verify_date_param_MongoDBSearch
   * F: Fn_SEARCH::verify_date_param_PurePerlSearch
   * F: Fn_SEARCH::verify_formQuery3_ForkingSearch
   * F: Fn_SEARCH::verify_formQuery3_MongoDBQuery
   * F: Fn_SEARCH::verify_formQuery3_MongoDBSearch
   * F: Fn_SEARCH::verify_formQuery3_PurePerlSearch
   * F: Fn_SEARCH::verify_lc_field_fully_qualified_ForkingSearch
   * F: Fn_SEARCH::verify_lc_field_fully_qualified_MongoDBQuery
   * F: Fn_SEARCH::verify_lc_field_fully_qualified_MongoDBSearch
   * F: Fn_SEARCH::verify_lc_field_fully_qualified_PurePerlSearch
   * F: Fn_SEARCH::verify_multiple_order_fields_ForkingSearch
   * F: Fn_SEARCH::verify_multiple_order_fields_MongoDBQuery
   * F: Fn_SEARCH::verify_multiple_order_fields_MongoDBSearch
   * F: Fn_SEARCH::verify_multiple_order_fields_PurePerlSearch
   * F: Fn_SEARCH::verify_refQuery_ForkingSearch
   * F: Fn_SEARCH::verify_refQuery_MongoDBQuery
   * F: Fn_SEARCH::verify_refQuery_MongoDBSearch
   * F: Fn_SEARCH::verify_refQuery_PurePerlSearch
RegisterTests has 30 unexpected results (of 185):
   * F: RegisterTests::verify_rejectEvilContent_ApacheLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_rejectEvilContent_ApacheLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_rejectEvilContent_NoLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_rejectEvilContent_NoLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_rejectEvilContent_TemplateLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_rejectEvilContent_TemplateLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithForm_ApacheLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithForm_ApacheLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithForm_NoLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithForm_NoLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithForm_TemplateLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithForm_TemplateLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithoutForm_ApacheLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithoutForm_ApacheLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithoutForm_NoLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithoutForm_NoLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithoutForm_TemplateLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithPMWithoutForm_TemplateLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithForm_ApacheLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithForm_ApacheLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithForm_NoLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithForm_NoLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithForm_TemplateLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithForm_TemplateLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithoutForm_ApacheLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithoutForm_ApacheLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithoutForm_NoLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithoutForm_NoLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithoutForm_TemplateLoginManager_AllowLoginName_HtPasswdManager_TopicUserMapping
   * F: RegisterTests::verify_userTopicWithoutPMWithoutForm_TemplateLoginManager_DontAllowLoginName_HtPasswdManager_TopicUserMapping
HoistMongoDBsTests has 36 unexpected results (of 94):
   * F: HoistMongoDBsTests::test_formQuery3
   * F: HoistMongoDBsTests::test_hoistLcLHSField
   * F: HoistMongoDBsTests::test_hoistLcLHSLikeName
   * F: HoistMongoDBsTests::test_hoistLcLHSName
   * F: HoistMongoDBsTests::test_hoistLcRHSName
   * F: HoistMongoDBsTests::test_hoistLengthLHSName
   * F: HoistMongoDBsTests::test_hoistLengthLHSNameGT
   * F: HoistMongoDBsTests::test_hoistLengthLHSString
   * F: HoistMongoDBsTests::test_hoistLengthLHSString_false
   * F: HoistMongoDBsTests::test_hoist_ImplicitFormNameBUG
   * F: HoistMongoDBsTests::test_hoist_Item10323
   * F: HoistMongoDBsTests::test_hoist_Item10323_1
   * F: HoistMongoDBsTests::test_hoist_Item10323_2
   * F: HoistMongoDBsTests::test_hoist_Item10323_2_not
   * F: HoistMongoDBsTests::test_hoist_Item10515
   * F: HoistMongoDBsTests::test_hoist_concat
   * F: HoistMongoDBsTests::test_hoist_concat2
   * F: HoistMongoDBsTests::test_hoist_concat2_false
   * F: HoistMongoDBsTests::test_hoist_concat3
   * F: HoistMongoDBsTests::test_hoist_concat3_false
   * F: HoistMongoDBsTests::test_hoist_concat_false
   * F: HoistMongoDBsTests::test_hoist_d2n
   * F: HoistMongoDBsTests::test_hoist_d2n_value
   * F: HoistMongoDBsTests::test_hoist_d2n_valueAND
   * F: HoistMongoDBsTests::test_hoist_dateAndRelationship
   * F: HoistMongoDBsTests::test_hoist_maths
   * F: HoistMongoDBsTests::test_hoist_ref
   * F: HoistMongoDBsTests::test_hoist_ref2
   * F: HoistMongoDBsTests::test_hoist_ref3
   * F: HoistMongoDBsTests::test_hoist_ref4
   * F: HoistMongoDBsTests::test_hoist_ref4_longhand
   * F: HoistMongoDBsTests::test_hoist_ref4_or
   * F: HoistMongoDBsTests::test_hoist_ref_CREATEINFO_longhand
   * F: HoistMongoDBsTests::test_hoist_ref_TOPICINFO_longhand
   * F: HoistMongoDBsTests::test_hoist_ref_TOPICINFO_longhand_plus
   * F: HoistMongoDBsTests::test_hoist_ref_TOPICINFO_longhand_plus_WEBHome

where the hoist tests are because i changed the javascript dramatically but have not updated the tests.

strangely, this is on an ubuntu lxc setup where i'm not running the tests with -CSAD

-- SvenDowideit - 05 Nov 2012
Topic attachments
I Attachment Action Size Date Who Comment
Foswikirev_15825_MongoFailures.txttxt Foswikirev_15825_MongoFailures.txt manage 27.9 K 03 Nov 2012 - 07:08 PaulHarvey test failures
LocalSite.cfgcfg LocalSite.cfg manage 36.5 K 03 Nov 2012 - 06:14 PaulHarvey My MongoDB config
cluster_layout.pngpng cluster_layout.png manage 85.1 K 14 Oct 2010 - 22:51 PaulHarvey Cluster layout
mongo_cluster.pngpng mongo_cluster.png manage 69.1 K 15 Oct 2010 - 00:17 PaulHarvey Mongo components
mongodb-config.confconf mongodb-config.conf manage 1.1 K 14 Oct 2010 - 06:33 PaulHarvey  
mongodb-set1.confconf mongodb-set1.conf manage 1.1 K 14 Oct 2010 - 06:32 PaulHarvey  
mongodb-test.loglog mongodb-test.log manage 3543.9 K 09 Dec 2011 - 02:23 PaulHarvey 220 of 3868 tests fail (Foswikirev:13326)
Topic revision: r22 - 05 Nov 2012, SvenDowideit
 
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. see CopyrightStatement. Creative Commons License