[Chandler-dev] Chandler background full-text indexing

Andi Vajda vajda at osafoundation.org
Tue May 23 16:44:00 PDT 2006

For the sake of performance in the Chandler's main UI view, I added support 
for doing full-text indexing with PyLucene in the background.

By default, the UI view in Chandler is now setup to not do any PyLucene 
indexing during commit. Instead another view, the Lucene view, picks up the 
changes it sees every minute and adds them to the repository Lucene index in 
the background.

Running Chandler with indexing the old way is done with, yes you guessed it, 
a new command line flag: --indexer=foreground   (short flag name: -i)
Again, by default, --indexer is set to 'background' and Chandler sets the main 
UI view to use background indexing and starts the repository background
indexer upon startup.

By default, a view does foreground indexing, that is, PyLucene indexing is 
done during commit. If you wish to change a view to use the background 
indexer instead, use the new view.setBackgroundIndexed(True) API.

The background indexer runs every minute or so. This value is hardcoded. At 
some point we need to have support for user preferences and we can then tie 
that value in with them. If you're in a real hurry to have your stuff indexed 
in the background right away, you can use the repository.notifyIndexer() API.

PyLucene indexing is also considerably faster now. I realized that the index 
I/O part of it was thrashing around in the repository so the actual indexing 
now happens in memory before being written to repository all at once.
In-memory indexing shouldn't use too much memory since the amount of indexing 
done each time corresponds to only one repository version.

This change made indexing about three times faster.


More information about the chandler-dev mailing list