[Chandler-dev] Poll: What performance tools do you use?

[Bryan Stearns]

Answers inline.

Heikki Toivonen wrote:
> 1. Are regression bugs usable to you?
>   
No
> 2. Are trend graphs usable to you?
> http://builds.osafoundation.org/perf_data/trends.html
>   
Yes, to the extent that I can see change over time, and consistent 
variability between platforms. Also, this is the only place where it 
looks like we've made any progress.
> 3. Are the tables from which trend graphs are drawn from usable to you?
> For example
> http://builds.osafoundation.org/perf_data/detail_20070318_20070417.html
>   
No: too variable.
> 4. Are the full results for today usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070417.html
>   
No: too variable, sample size too small.
> 5. Are the daily graphs usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070417.html
>   
No. too variable, sample size too small.
> 6. Are historical daily reports usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070416.html
>   
No. too variable, sample size too small.
> 7. Are the deltas and std.dev usable to you on
> http://builds.osafoundation.org/perf_data/tbox.html
>   
No, too variable, misleading color coding of both time and delta boxes. 
Std devs ridiculously large (which is not a problem of presentation: 
it's a problem of measurement).
> 8. If you would like to change the colors or format of
> http://builds.osafoundation.org/perf_data/tbox.html please list the
> changes here:
Until the perf tests produce more reliable timings, I don't think I'll 
need this for anything.

It's too depressing to look at in detail: I block it out when looking at 
the tinderbox page. I really only use it to get to the "trends" link in 
the top left corner.

> 9. Is rt.py -p usable to you?
>   
I've only used it to verify that I haven't broken anything before 
checking in a change to a perf test. I don't run it before normal checkins.

> 10. Is rt.py -t usable to you?
>   
Yes, when perf testing and investigating failures of normal tests.

When looking at performance, I use it to run a specific perf test, only 
to see what's being executed by that test; I then manually add 
hotshotting to get statistics about particular parts of the code.

I usually run it with --verbose, so I can capture the command line it's 
using to start Chandler. I then use that command line (with --indexer 
none) to get timings from the hotshot calls I've added. I never look at 
the numbers other than hotshot (so absolute time doesn't mean anything: 
I'm not on typical hardware, I'm running with optimization off so that I 
can debug, etc).

> 11. Is rt.py -P usable to you?
>   
No.
> 12. Is rt.py --repeat usable to you?
>   
No.
> 13. Any other things you can think of that either add value or are
> irrelevant or you would like changed:
>
>
>