[Chandler-dev] Poll: What performance tools do you use?

Grant Baillie grant at osafoundation.org
Tue Apr 17 17:18:32 PDT 2007

On 17 Apr, 2007, at 13:47, Heikki Toivonen wrote:

> Only Bryan and Andi have answered to some of the questions I asked  
> in my
> emails yesterday, so trying again in a poll format. I'd like to know
> what tools and procedures we currently have in place add value, and
> which do not. Based on this I can then drop those procedures and  
> reports
> that aren't cost effective to do.
> If something is just nice to look at, like trend graph, but you don't
> actually do anything with the information, then I'd like you to answer
> in the negative for that case.
> You can answer privately or on the list; I'll summarize the answers by
> Thursday evening the latest.
> 1. Are regression bugs usable to you?

Not really: Performance fixes are in general incremental, and rarely  
involve "fixing" a particular regression. So, the bugs tend to float  
around getting retargetted from release to release, and occasionally  
when a perf # gets good enough, they bug gets closed or moves over to  

However, I think that the information as to when a perf regression  
occurs is very useful, since it helps point to where we can improve  
performance in the future. (As you know, the perf tests aren't  
exactly run against every revision, and there can be pretty large  
variability in a single test, so it's not always possible, based on  
the tinderbox numbers, to pin down a problem to a single revision).

I'm not sure, though, what the best way to record this information,  
and associated conversations between developers, is.

> 2. Are trend graphs usable to you?
> http://builds.osafoundation.org/perf_data/trends.html
> 3. Are the tables from which trend graphs are drawn from usable to  
> you?
> For example
> http://builds.osafoundation.org/perf_data/ 
> detail_20070318_20070417.html
> 4. Are the full results for today usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070417.html
> 5. Are the daily graphs usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070417.html
> 6. Are historical daily reports usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070416.html

[2--6] I do look at these occasionally, but not regularly. Having the  
graphs makes it easier to tell where Things (Good and Bad) Happened,  
but probably not essential.

> 7. Are the deltas and std.dev usable to you on
> http://builds.osafoundation.org/perf_data/tbox.html

I understand them in the context or rt.py --repeat. But the ones in  
tinderbox data, are the standard deviations taken across different  
revisions? If so, they're not very good numbers statistically. In  
addition, the deltas don't make sense to me: we shouldn't be  
comparing a single revision to itself, for example. Lastly (as I  
think you mentioned in an earlier email), clearing out the data at  
the beginning of each day, instead of tracking the previous 24 hours,  
makes sense. (I've had at least one case where I never got to see the  
nice green improvement from a perf fix because it was checked in  
around 10pm).

> 8. If you would like to change the colors or format of
> http://builds.osafoundation.org/perf_data/tbox.html please list the
> changes here:
> 9. Is rt.py -p usable to you?

Yup ... I use it all the time.

> 10. Is rt.py -t usable to you?

I use it, although often in conjunction with --dry-run so that I can  
use pdb on the "real" test invocation.

As an aside: Why is "-t" even an option? Couldn't rt.py just treat  
all arguments (i.e. things that aren't options on the command line)  
as tests to be run?

> 11. Is rt.py -P usable to you?

Lately, when I've tried to use it, hotshot has not been happy with  
loading the profile afterwards. So, I've typically switched to insert  
hotshot.runcall in the source(*), or sometimes util.easyprof

(*) But you knew that already :)

> 12. Is rt.py --repeat usable to you?

Yes, I use it quite often. E.g. if I'm trying to figure out if a  
patch really makes a difference to performance, I'll run it  
(sometimes overnight) with some number of --repeats, so as to  
minimize uncertainty of crazy out-of-range values.

> 13. Any other things you can think of that either add value or are
> irrelevant or you would like changed:
> -- 
>   Heikki Toivonen
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev

More information about the chandler-dev mailing list