[Chandler-dev] No more perf regression bugs?

[Philippe Bossut]

Hi,

I'm jumping in that thread late and wanted to stress some points from a 
project management standpoint:

- We care about performance: right now, poor performance is the biggest 
recognized issue for Chandler's adoption. This is in discussing this 
that we decided to devote 3 weeks of the entire team working on this 
matter alone, trying to understand what can be done, make headways in 
some places, allow devs to work collaboratively on some issues, etc... 
We'll have more work to do on performance after those 3 weeks for sure 
but the objective was to have everyone involved in it and caring for it 
for a while. The message is: everyone owns performance.

- Baseline and regression are two different things: one aspect of the 
discussion that started the thread but was somewhat muddled afterward is 
the difference between "baseline performance" and "performance 
regression", resulting in people talking pass each other. Both are 
important in their own right though, clearly, a "baseline" must be 
reached before we can consider "regressions". This is why we modified 
the performance matrix so that scenarios that do not reach the 
"acceptable" level yet are clearly identified, regardless on how best or 
worse they compared to the previous run. It's true (as stated by 
stearns) that for Preview, we'll likely aim for "acceptable" and not get 
to "ideal", leaving the matrix in an all-orange state. Let's be clear 
here: orange is good, we like orange.

- Regression tracking is important moving forward: we'll need to have a 
performance tracking system so that we get alerted when performance of 
the base scenarios is falling behind for *any* reason. Everything that 
has been said in the thread is true  and fair and what needs to be done 
when a regression is detected can be debated (log a bug, send an email, 
rollback a change, chat on IRC, etc...) and we need to find our own 
Chandler way of doing it. One thing is sure though: if not monitored and 
cared for, performance will fall behind. That's a law of software 
development...

- Bugzilla is our tool to track incidents : this came up during the 
discussion. Bugzilla is not perfect, it has issues but, as a team, we 
need to use one single tracking tool and this is the one we use in that 
team right now. We even use it to track feature work (tasks) at least in 
the Apps team. A perf drop is a bona fide "incident" and using Bugzilla 
to eventually track it seems  fine. In any case, the "tracking perf 
regression" issue is orthogonal to the "is Bugzilla an adequate tracking 
system" issue. That's something we may want to discuss on its own thread.

In conclusion, considering the current objective (get to baseline) and 
to come back to the title of the thread, I'm voting for a moratorium on 
the current "log a regression bug" policy, at least till we reach 
"baseline" (which may happen *after* the end of the current perf 
tweaking period).

In the meantime, we can discuss of what the perf regression alert policy 
should be.

Cheers,
- Philippe