Item10532: WIBNIF we allowed queries to go to slaves (slave_ok)

mongod seems to process requests one-at-a-time (at least, JS queries do anyway).

The failure-mode is quite interesting: the entire foswiki site stops responding until it has caught up. We had some bots crawling the site and yandex seemed to keep hitting a long-running (~20s) ref query again and again. For as long as it was doing this, it seems other queries had little opportunity to execute. And most other queries are on the order of 0.05s or less.

So, we have plenty of VMs, and we already run replica sets. It seems as if it might be trivial to allow Foswiki to query any of the replica members rather than just the master. MongoDB::Cursor talks about the slave_ok.

The other way of dividing the work is to use shards, but I suspect that exploring tuning parameters for a decent shard key and measuring the result (perf. artifacts from querying, sorting, merging results from multiple shards) will be complicated and time-consuming to verify (and if everybody is hitting data from a single shard then it's of no benefit anyway)...

-- PaulHarvey - 24 Mar 2011

commited, tryit

-- SvenDowideit - 28 Mar 2011

hrmm. Must need more than what we've done... I have four members in the set, but only the primary was being queried. Also, tried setting up a mongos daemon for foswiki to talk to - no difference.

-- PaulHarvey - 30 Mar 2011

This will require more code: the perl driver doesn't automatically distribute queries, it's up to the application to discover the replSet masters members & open connections to them.

-- PaulHarvey - 03 Apr 2011

This might be reslved now - you need to list each host in the mongoDBPlugin host setting

-- SvenDowideit - 19 Apr 2011

I'm using

$Foswiki::cfg{MongoDBPlugin}{host} = 'a.server.org:27017,b.server.org:27017,c.server.org:27017';
$Foswiki::cfg{MongoDBPlugin}{auto_reconnect} = 1;
$Foswiki::cfg{MongoDBPlugin}{auto_connect} = 1;
$Foswiki::cfg{MongoDBPlugin}{timeout} = 20000;
$Foswiki::cfg{MongoDBPlugin}{find_master} = 1;
$Foswiki::cfg{MongoDBPlugin}{w} = 1;
$Foswiki::cfg{MongoDBPlugin}{wtimeout} = 1000;
$Foswiki::cfg{Plugins}{MongoDBPlugin}{ExperimentalCode} = 1;

Which still only pegs whoever happens to be the PRIMARY @ 100% CPU (the rest remain idle).

Happily, if I kill the PRIMARY, Foswiki is still able to take writes & server queries. Note: there still needs to be a majority of remaining replSet members up for the vote to work

-- PaulHarvey - 20 Apr 2011

Commenting is disabled while not logged in