[Dev] Performance tests

[Ted Leung]

This all sounds fine to me.   Do we want to do a similar thing for 
footprint?

Ted

On Nov 9, 2004, at 5:12 PM, Heikki Toivonen wrote:

> We'd like to get some performance tests going during the 0.5 
> timeframe. Here are my thoughts on how to go about this.
>
> First of all, I think performance tests should be run automatically by 
> tinderbox as part of normal test cycle. There are some requirements 
> before this can work reliably, though.
>
> In order to run reliable performance tests on the build machines we 
> need to make sure nothing else is done on those machines since that 
> would immediately skew the results. Before this can happen we need an 
> automated tool to create binary tarballs from external and internal - 
> I wrote about this yesterday. I have a script pretty close to testing. 
> We'd also need two more build machines - one Mac and one Windows to do 
> full builds.
>
> At a minimum, and as a first step, we should add performance specific 
> unit tests. (NOTE: Our unit tests already report the time it took to 
> run them - look at the long logs from tinderbox.) Some tests I would 
> like to see include: parcel loading, startup, view switching, and 
> various repository specific tests some others can articulate better. 
> Performance tests need to be run several times (3-5 for tests at a 
> minimum for something that takes non-trivial time, otherwise hundreds 
> or thousands of iterations might be prudent), and report things like 
> minimum time, maximum time, average, median. Naturally, we should 
> concentrate on the performance of the optimized bits, not debug 
> versions.
>
> Making the results of the performance tests more accessible would be 
> the next step. Tinderbox server should extract the performance results 
> from the full logs, and display at a glance results along with the I, 
> L, C links in the build cells. Like: "P=334.2s" could mean parcel 
> loader test passed in 334.2 seconds.
>
> Next  step would be preserving the historical data so that we can 
> easily see trends. The tinderbox server should log performance 
> results, and be able to present it in various formats (trend graphs, 
> comma separates values, ...) These should be available as links from 
> the short forms on tbox.
>
> This would give us the tools to continuously monitor our progress.
>
> However, I also think we need to run performance tests in an ad hoc 
> manner on our reference systems (qa lab machines) to determine 
> acceptable performance (it  wouldn't help much if our multiprocessor, 
> performance optimized build machines launched Chandler in 1 second 
> while it took a minute on typical desktop). Summaries of all these 
> tests and our progress and plans should be reported on the wiki.
>
> -- 
>   Heikki Toivonen
>
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/dev
>