Item10994: MongoDBPlugin can run out of indexes before they're enabled on _topic
, _web
and friends
Priority: Urgent
Current State: Closed
Released In: n/a
Target Release: n/a
Background:
- Sometimes imports don't run to completion. Maybe I interrupt a web that's only half-loaded. Could this be related to indexes not being set on the next run?
- Seems more reliable if I purge mongo completely before re-importing. But this doesn't guarantee a good import (somtimes end up with no indexes set, even from a clean mongodb)
Here's an example of what happened on an otherwise normal-looking import of
Marine/SeaSlugs/Messages
web:
> db.map.find({_id : "Marine/SeaSlugs/Messages"})
{ "hash" : "web_60f1085a95ad14944fee4a10fc8a1a34", "_id" : "Marine/SeaSlugs/Messages" }
> use web_60f1085a95ad14944fee4a10fc8a1a34
switched to db web_60f1085a95ad14944fee4a10fc8a1a34
> show collections
ACLProfiles
current
system.indexes
system.js
system.profile
> db.current.getIndexes()
[
{
"name" : "_id_",
"ns" : "web_60f1085a95ad14944fee4a10fc8a1a34.current",
"key" : {
"_id" : 1
},
"v" : 0
},
{
"ns" : "web_60f1085a95ad14944fee4a10fc8a1a34.current",
"key" : {
"FIELD.MessageID.value" : NumberLong(1)
},
"name" : "FIELD.MessageID",
"background" : true,
"v" : 0
}
]
>
Yes, I forgot to set MONITOR_INDEXES this time. Here is the output from that web, anyway (using
-web all -recurse on -fork 1
):
start web: Marine/SeaSlugs/Messages (36586) -> web_60f1085a95ad14944fee4a10fc8a1a34
imported 1000
imported 2000
imported 3000
imported 4000
imported 5000
imported 6000
imported 7000
imported 8000
imported 9000
imported 10000
imported 11000
imported 12000
imported 13000
imported 14000
imported 15000
imported 16000
imported 17000
imported 18000
imported 19000
imported 20000
imported 21000
imported 22000
imported 23000
imported 24000
imported 25000
imported 26000
imported 27000
imported 28000
imported 29000
imported 30000
imported 31000
imported 32000
imported 33000
imported 34000
imported 35000
imported 36000
imported 36587.14
Running again with monitor_index
--
PaulHarvey - 22 Jul 2011
considering that i actually call ensureIndex on
every updateTopic call, because updates are much slower when there are no indexes, I really don't know - its basically bizzare.
--
SvenDowideit - 22 Jul 2011
Seems that CREATEINFO, TOPICINFO, _web and _topic aren't getting indexes set.
So I've updated
https://gist.github.com/941875
var databases = db.adminCommand('listDatabases')['databases'];
for (var item in databases) {
db = db.getMongo().getDB(item['name']);
db.current.ensureIndex({_web: 1});
db.current.ensureIndex({_topic: 1});
db.current.ensureIndex({'TOPICINFO.author': 1})
db.current.ensureIndex({'TOPICINFO.date': 1})
db.current.ensureIndex({'CREATEINFO.author': 1})
db.current.ensureIndex({'CREATEINFO.date': 1})
db.current.ensureIndex({'TOPICPARENT.name': 1})
db.current.ensureIndex({'FORM.name': 1})
}
--
PaulHarvey - 12 Aug 2011
Re-titled: this occurs when there are too many DataForm fields, we use up all the indexes on those first, before the important ones get set.
Failure mode of not having indexes set on
_web
or
_topic
etc. is that the default sorting will fail on large webs that can't be sorted in-memory by MongoDB.
--
PaulHarvey - 15 Mar 2012
Finally done, I think. Even added a test!
--
PaulHarvey - 16 Mar 2012
Re-opened to squelch the log noise down a bit (should only warn once per web-fieldname)
--
PaulHarvey - 26 Mar 2012