[Cosmo-dev] Performance test tool requirements

[Mikeal Rogers]

The next release is approaching, and we're beginning to do some  
initial performance work so this seems like a good time to begin the  
discussion about requirements for QA development on the test tools to  
support the next release.

Our current test tool suite (currently being called 'broadsword') is  
about to go 0.1, but they currently don't contain any of the  
necessary plumbing for performance analysis. We now need to gather  
requirements for 0.2 in order to be ready in time to write new  
performance tests and start analysis by the time QA gets handed the  
next release.

We need to know the following;

1) What tests will need to be measured (We can handle writing the  
development requirements needed to fulfill the test requirements)
2) What format we want the performance analysis displayed (pretty  
website, pretty graphics, text in an email?)
3) What our target hardware & platform is (we need to buy this and  
begin to setup the test farm)

There are obviously many other tests and metrics that we'll need for  
the hosted service, and although not all of that work is targeted for  
the next release we'd like to begin to get an idea of it so that we  
don't architect things in such a way that it's difficult to add any  
necessary test tool features later.

-Mikeal