[Chandler-dev] Chandler background full-text indexing

[John Anderson]

This seems like another example of premature generalization -- I think 
it would be better to add the option when we know be have a need for it 
rather than add it in case we might someday need it.

Andi Vajda wrote:
>
> On Tue, 23 May 2006, John Anderson wrote:
>
>> Ok, I missunderstood -- I'm glad I don't need to add another option 
>> to my command line. But how about getting rid of the old behavior and 
>> the obscure option? Is there anyone who plans to use it?
>
> The old behaviour is more predictable from an execution flow 
> standpoint. There may be situations, such as performance testing, 
> where background indexing may be in the way. I don't know if that is a 
> good example, but having that
> possibility seemed like a reasonable thing to have.
>
> Andi..
>
>>
>> John
>>
>> Andi Vajda wrote:
>>>
>>> On Tue, 23 May 2006, John Anderson wrote:
>>>
>>>> Is there any reason we don't want to change the default to index in 
>>>> the background -- or maybe better yet, only index in the background 
>>>> and get rid of another obscure option. In that case the UI code 
>>>> could optionally force indexing if it became necessary.
>>>
>>> The obscure option is the default. No need to use it unless you want 
>>> the old behaviour, that is NOT to index in the background.
>>>
>>> Andi..
>>>
>>>>
>>>> Andi Vajda wrote:
>>>>>
>>>>> For the sake of performance in the Chandler's main UI view, I 
>>>>> added support for doing full-text indexing with PyLucene in the 
>>>>> background.
>>>>>
>>>>> By default, the UI view in Chandler is now setup to not do any 
>>>>> PyLucene indexing during commit. Instead another view, the Lucene 
>>>>> view, picks up the changes it sees every minute and adds them to 
>>>>> the repository Lucene index in the background.
>>>>>
>>>>> Running Chandler with indexing the old way is done with, yes you 
>>>>> guessed it, a new command line flag: --indexer=foreground   (short 
>>>>> flag name: -i)
>>>>> Again, by default, --indexer is set to 'background' and Chandler 
>>>>> sets the main UI view to use background indexing and starts the 
>>>>> repository background
>>>>> indexer upon startup.
>>>>>
>>>>> By default, a view does foreground indexing, that is, PyLucene 
>>>>> indexing is done during commit. If you wish to change a view to 
>>>>> use the background indexer instead, use the new 
>>>>> view.setBackgroundIndexed(True) API.
>>>>>
>>>>> The background indexer runs every minute or so. This value is 
>>>>> hardcoded. At some point we need to have support for user 
>>>>> preferences and we can then tie that value in with them. If you're 
>>>>> in a real hurry to have your stuff indexed in the background right 
>>>>> away, you can use the repository.notifyIndexer() API.
>>>> I'm surprised you didn't add another command line argument to 
>>>> specify the interval :)
>>>>>
>>>>> PyLucene indexing is also considerably faster now. I realized that 
>>>>> the index I/O part of it was thrashing around in the repository so 
>>>>> the actual indexing now happens in memory before being written to 
>>>>> repository all at once.
>>>>> In-memory indexing shouldn't use too much memory since the amount 
>>>>> of indexing done each time corresponds to only one repository 
>>>>> version.
>>>>>
>>>>> This change made indexing about three times faster.
>>>>>
>>>>> Andi..
>>>>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>>>>
>>>>> Open Source Applications Foundation "chandler-dev" mailing list
>>>>> http://lists.osafoundation.org/mailman/listinfo/chandler-dev
>>>>
>>