[Dev] 0.7 perf goals

[Andi Vajda]

On Tue, 6 Dec 2005, Bryan Stearns wrote:

> It's important to run the tests against a consistent "base" repository, 
> whether it's empty or not, and it's important to build that repository using 
> the same sources you're using for Chandler (so, saving off built repositories 
> to use across Chandler versions would scare me).

Yes, definitely, consistency is very important or else all comparisons are 
bogus. My point had to do with the potential expendability of non-real end 
user situations when compared to real end user situations.
As for the across chandler versions issue, we sure have to be able to attain 
that goal some day.

> There's "dynamic state" in the repository that strongly influences 
> performance: collection indexes and cached block trees come to mind 
> immediately, and there's probably more; block tree caching has already caused 
> confusion in interpreting performance measurements.

Until that is resolved, we ought to be very careful with comparisons.

> It might mean that running the suite means building a specific-sized 
> repository using current sources, then running Chandler against a fresh copy 
> of it for each test (or batch of related tests where we'll accept the 
> possibility of the tests influencing each other).  We might even build 
> several test repositories this way (empty, 100, 1000, 10000 items?) and 
> choose an appropriate-sized one for each test batch so that the execution 
> time is acceptable (say, shooting for a test runtime of less than a minute to 
> keep the whole suite manageable).

Or start with building the repository to some consistent specifications of 
data, then back it up and restore it before each test is run. Some of our 
calendar tests, for instance, are dependant on the date they are run since, 
depending on that date, different events are relevant.

Andi..