Item5647: Make the indexing process more robust
Priority: Urgent
Current State: Being Worked On
Released In:
Target Release: n/a
A crash in indexing one file or one topic should not lead to a stop of the whole indexing process.
--
MarkusHesse - 21 May 2008
Re-naming the plugin, and needs to check if this still applies.
AndrewJones did a lot of work to improve robustness, but I'm not sure about a crash of a single file. He tried to remove crashes all together, but it needs testing...
--
OlivierRaginel - 25 Sep 2009
i can confirm this can still happen. strangely, the
ks_test
was able to stringify the spreadsheet (about 18MB) in a couple of minutes, but
kinoindex
didn't manage to process the same file, even overnight.
--
WillNorris - 27 Feb 2010
Most probably your xls file is password protected. There are no third-party libraries to even detect that an xls is password protected. Instead the reader will consume it as is and generate large internal data structures while reading some arbitrary bits.
Been addressed in
StringifierContrib, sort of, by creating a separate cmdline perl to real the xls. If it crashes, it does not take
the rest of the indexer with it.
--
MichaelDaum - 27 Feb 2010
well, i don't think it was password-protected as
ks_test
was able to parse and produce results for the same file which was hanging the indexer. so, that's very disconcerting. also, it's not about the indexer crashing, but instead hanging. so, the wrapper will need some sort of watchdog capability. i'm sure
StringifierContrib will provide some inspiration, though the long term goal is to
refactor KinoSearchContrib to use StringifierContrib.
--
WillNorris - 11 Mar 2010