[Dev] Performance tests

[Heikki Toivonen]

We'd like to get some performance tests going during the 0.5 timeframe. 
Here are my thoughts on how to go about this.

First of all, I think performance tests should be run automatically by 
tinderbox as part of normal test cycle. There are some requirements 
before this can work reliably, though.

In order to run reliable performance tests on the build machines we need 
to make sure nothing else is done on those machines since that would 
immediately skew the results. Before this can happen we need an 
automated tool to create binary tarballs from external and internal - I 
wrote about this yesterday. I have a script pretty close to testing. 
We'd also need two more build machines - one Mac and one Windows to do 
full builds.

At a minimum, and as a first step, we should add performance specific 
unit tests. (NOTE: Our unit tests already report the time it took to run 
them - look at the long logs from tinderbox.) Some tests I would like to 
see include: parcel loading, startup, view switching, and various 
repository specific tests some others can articulate better. Performance 
tests need to be run several times (3-5 for tests at a minimum for 
something that takes non-trivial time, otherwise hundreds or thousands 
of iterations might be prudent), and report things like minimum time, 
maximum time, average, median. Naturally, we should concentrate on the 
performance of the optimized bits, not debug versions.

Making the results of the performance tests more accessible would be the 
next step. Tinderbox server should extract the performance results from 
the full logs, and display at a glance results along with the I, L, C 
links in the build cells. Like: "P=334.2s" could mean parcel loader test 
passed in 334.2 seconds.

Next  step would be preserving the historical data so that we can easily 
see trends. The tinderbox server should log performance results, and be 
able to present it in various formats (trend graphs, comma separates 
values, ...) These should be available as links from the short forms on 
tbox.

This would give us the tools to continuously monitor our progress.

However, I also think we need to run performance tests in an ad hoc 
manner on our reference systems (qa lab machines) to determine 
acceptable performance (it  wouldn't help much if our multiprocessor, 
performance optimized build machines launched Chandler in 1 second while 
it took a minute on typical desktop). Summaries of all these tests and 
our progress and plans should be reported on the wiki.

-- 
   Heikki Toivonen


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url : http://lists.osafoundation.org/pipermail/dev/attachments/20041109/5073a0bc/signature.bin