[Chandler-dev] Chandler background full-text indexing

[John Anderson]

Is there any reason we don't want to change the default to index in the 
background -- or maybe better yet, only index in the background and get 
rid of another obscure option. In that case the UI code could optionally 
force indexing if it became necessary.

Andi Vajda wrote:
>
> For the sake of performance in the Chandler's main UI view, I added 
> support for doing full-text indexing with PyLucene in the background.
>
> By default, the UI view in Chandler is now setup to not do any 
> PyLucene indexing during commit. Instead another view, the Lucene 
> view, picks up the changes it sees every minute and adds them to the 
> repository Lucene index in the background.
>
> Running Chandler with indexing the old way is done with, yes you 
> guessed it, a new command line flag: --indexer=foreground   (short 
> flag name: -i)
> Again, by default, --indexer is set to 'background' and Chandler sets 
> the main UI view to use background indexing and starts the repository 
> background
> indexer upon startup.
>
> By default, a view does foreground indexing, that is, PyLucene 
> indexing is done during commit. If you wish to change a view to use 
> the background indexer instead, use the new 
> view.setBackgroundIndexed(True) API.
>
> The background indexer runs every minute or so. This value is 
> hardcoded. At some point we need to have support for user preferences 
> and we can then tie that value in with them. If you're in a real hurry 
> to have your stuff indexed in the background right away, you can use 
> the repository.notifyIndexer() API.
I'm surprised you didn't add another command line argument to specify 
the interval :)
>
> PyLucene indexing is also considerably faster now. I realized that the 
> index I/O part of it was thrashing around in the repository so the 
> actual indexing now happens in memory before being written to 
> repository all at once.
> In-memory indexing shouldn't use too much memory since the amount of 
> indexing done each time corresponds to only one repository version.
>
> This change made indexing about three times faster.
>
> Andi..
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev