Item5647: Make the indexing process more robust

pencil
Priority: Urgent
Current State: Being Worked On
Released In:
Target Release: n/a
Applies To: Extension
Component: KinoSearchContrib
Branches:
Reported By: MarkusHesse
Waiting For: Main.WillNorris
Last Change By: WillNorris
A crash in indexing one file or one topic should not lead to a stop of the whole indexing process.

-- MarkusHesse - 21 May 2008

Re-naming the plugin, and needs to check if this still applies.

AndrewJones did a lot of work to improve robustness, but I'm not sure about a crash of a single file. He tried to remove crashes all together, but it needs testing...

-- OlivierRaginel - 25 Sep 2009

i can confirm this can still happen. strangely, the ks_test was able to stringify the spreadsheet (about 18MB) in a couple of minutes, but kinoindex didn't manage to process the same file, even overnight.

-- WillNorris - 27 Feb 2010

Most probably your xls file is password protected. There are no third-party libraries to even detect that an xls is password protected. Instead the reader will consume it as is and generate large internal data structures while reading some arbitrary bits.

Been addressed in StringifierContrib, sort of, by creating a separate cmdline perl to real the xls. If it crashes, it does not take the rest of the indexer with it.

-- MichaelDaum - 27 Feb 2010

well, i don't think it was password-protected as ks_test was able to parse and produce results for the same file which was hanging the indexer. so, that's very disconcerting. also, it's not about the indexer crashing, but instead hanging. so, the wrapper will need some sort of watchdog capability. i'm sure StringifierContrib will provide some inspiration, though the long term goal is to refactor KinoSearchContrib to use StringifierContrib.

-- WillNorris - 11 Mar 2010

ItemTemplate edit

Summary Make the indexing process more robust
ReportedBy MarkusHesse
Codebase
SVN Range TWiki-5.0.0, Sun, 04 May 2008, build 16770
AppliesTo Extension
Component KinoSearchContrib
Priority Urgent
CurrentState Being Worked On
WaitingFor WillNorris
Checkins KinoSearchContrib:de31b5ea4b05 distro:ee959792d74f
TargetRelease n/a
ReleasedIn
Topic revision: r8 - 18 Mar 2010, WillNorris
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. See Copyright Statement. Creative Commons License    Legal Imprint    Privacy Policy