[Dev] 0.7 perf goals

[Katie Capps Parlante]

Heikki Toivonen wrote:
> There are a couple of approaches we could take. One is to concentrate on
> the cases that are furthest from our 0.6 goals. The other is to take a
> broad look and work on all the cases that are not yet under the ideal
> limits. In the first case our numeric goals would remain unchanged. In
> the second case we would set new, tighter goals for the cases that did
> make it under acceptable limits but are not yet under ideal limits.
> Personally I am in favor of the latter. I have some initial suggestions
> here:
> http://wiki.osafoundation.org/bin/view/Journal/ZeroSevenPerfGoals20051204

Another approach we could take is to increase the size of the 
collections (and repository) that we expect to handle. Instead of a 3000 
item calendar, we could say a 10,000 item repository, with several 
collections/calendars. Nailing down the tenets might help us pick a good 
target, but the general goal here would be to ratchet up the size of the 
repository. We could combine this with focus on a specific set of cases 
and/or ratcheting down the response times.

> There's also a question of policy. Do we want to mandate a strict policy
> of not allowing performance regressions (regressions backed out, no
> questions asked), or do we consider each regression on a case by case
> basis? The danger with the second approach is that if we leave the
> regression in, and other code starts piling on top of those changes, it
> can be hard to back out even if we wanted to. Personally I am slightly
> in favor of a strict policy, but the other approach might also work well
> if we have just a few regressions.
> 
> Mozilla has long maintained a very strict policy where regressions are
> simply backed out. If you really wanted your fix/feature in, there were
> two options: make a change such that performance does not regress, or
> convince everyone that your change is so important it trumps performance.

This approach seems pretty reasonable for Mozilla (or Safari, which I 
hear has a similar system), where the intense focus is on a particular 
well-known use case. In our case, we're continuing to tweak the 
functionality and even change the formulation of the use cases that we 
measure. I think it would be overly constraining at this point to have a 
formal "back all regressions out" policy.

Cheers,
Katie