[Dev] 0.7 perf goals

[Bryan Stearns]

It's important to run the tests against a consistent "base" repository, 
whether it's empty or not, and it's important to build that repository 
using the same sources you're using for Chandler (so, saving off built 
repositories to use across Chandler versions would scare me).

There's "dynamic state" in the repository that strongly influences 
performance: collection indexes and cached block trees come to mind 
immediately, and there's probably more; block tree caching has already 
caused confusion in interpreting performance measurements.

It might mean that running the suite means building a specific-sized 
repository using current sources, then running Chandler against a fresh 
copy of it for each test (or batch of related tests where we'll accept 
the possibility of the tests influencing each other).  We might even 
build several test repositories this way (empty, 100, 1000, 10000 
items?) and choose an appropriate-sized one for each test batch so that 
the execution time is acceptable (say, shooting for a test runtime of 
less than a minute to keep the whole suite manageable).

...Bryan

Heikki Toivonen wrote:

>Andi Vajda wrote:
>  
>
>>I was thinking about all the perf tests that start from an empty
>>repository.
>>    
>>
>
>I considered the usefulness of these tests some while back, with the
>hope of being able to eliminate them, but decided to leave them. I think
>they help track down and fix the causes of the fixed perf costs, i.e.
>costs that are not dependent on the size of the repository. For example,
> creating new event even with empty repository takes about half a second
>(compared with up to 4 seconds with 3k events). For both of those cases
>the ideal goal is <0.1s, and to get there the fixed costs must still
>come down.
>
>If nobody else thinks those tests are useful we could stop running them.
>Opinions?
>
>  
>
>------------------------------------------------------------------------
>
>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
>Open Source Applications Foundation "Dev" mailing list
>http://lists.osafoundation.org/mailman/listinfo/dev
>  
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/dev/attachments/20051206/db0487cc/attachment.htm