Item5647: Make the indexing process more robust

A crash in indexing one file or one topic should not lead to a stop of the whole indexing process.

-- MarkusHesse - 21 May 2008

Re-naming the plugin, and needs to check if this still applies.

AndrewJones did a lot of work to improve robustness, but I'm not sure about a crash of a single file. He tried to remove crashes all together, but it needs testing...

-- OlivierRaginel - 25 Sep 2009

i can confirm this can still happen. strangely, the ks_test was able to stringify the spreadsheet (about 18MB) in a couple of minutes, but kinoindex didn't manage to process the same file, even overnight.

-- WillNorris - 27 Feb 2010

Most probably your xls file is password protected. There are no third-party libraries to even detect that an xls is password protected. Instead the reader will consume it as is and generate large internal data structures while reading some arbitrary bits.

Been addressed in StringifierContrib, sort of, by creating a separate cmdline perl to real the xls. If it crashes, it does not take the rest of the indexer with it.

-- MichaelDaum - 27 Feb 2010

well, i don't think it was password-protected as ks_test was able to parse and produce results for the same file which was hanging the indexer. so, that's very disconcerting. also, it's not about the indexer crashing, but instead hanging. so, the wrapper will need some sort of watchdog capability. i'm sure StringifierContrib will provide some inspiration, though the long term goal is to refactor KinoSearchContrib to use StringifierContrib.

-- WillNorris - 11 Mar 2010