[Chandler-dev] Poll: What performance tools do you use?

Andi Vajda vajda at osafoundation.org
Tue Apr 17 14:26:01 PDT 2007


On Tue, 17 Apr 2007, Heikki Toivonen wrote:

> 1. Are regression bugs usable to you?

See my replies to you yesterday. If the regression is real and you've talked 
to me first, then yes, the bug is real and usable.

> 2. Are trend graphs usable to you?
> http://builds.osafoundation.org/perf_data/trends.html

Sort of, they seem to only make sense in the long run.

> 3. Are the tables from which trend graphs are drawn from usable to you?
> For example
> http://builds.osafoundation.org/perf_data/detail_20070318_20070417.html

No. The numbers are all over the map.

> 4. Are the full results for today usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070417.html

No. The numbers are all over the map.

> 5. Are the daily graphs usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070417.html

No.

> 6. Are historical daily reports usable to you? For example
> http://builds.osafoundation.org/perf_data/detail_20070416.html

No, the interval is too short and the numbers are all over the map.

> 7. Are the deltas and std.dev usable to you on
> http://builds.osafoundation.org/perf_data/tbox.html

No. The numbers are all over the map, even when comparing a version against 
itself.

> 8. If you would like to change the colors or format of
> http://builds.osafoundation.org/perf_data/tbox.html please list the
> changes here:

No change.

> 9. Is rt.py -p usable to you?

No, the profiles that come out don't load in hotshot.

> 10. Is rt.py -t usable to you?

Dunno, I still use do_tests.sh and RunPython directly.
I saw no need to move to rt.py. Maybe there is one, I just don't know about 
it.

> 11. Is rt.py -P usable to you?

Err, maybe my answer for 'rt -p' is the one applicable here.
I guess, I don't know what 'rt -p' does. Run the perf tests maybe ?
I never do, their numbers only make sense over a week or so to get a trend.
Not for a punctual timing. For that, I've only seen direct instrumenting to be 
reliable.

> 12. Is rt.py --repeat usable to you?

Didn't know it existed. See my -P answer.

> 13. Any other things you can think of that either add value or are
> irrelevant or you would like changed:

The perf tboxes are too random to be useful for anything beyond detecting a 
trend over a week or so of runs.

Andi..


More information about the chandler-dev mailing list