[Cosmo-dev] Cosmo release frequency

[Mikeal Rogers]

On Oct 8, 2007, at 3:20 PM, Aparna Kadakia wrote:

> Chiming in to bring in the QA perspective on both week v/s month  
> long release cycles.
>
> While I am in favor of weekly releases, it does put certain strain  
> on the under-staffed QA team to qualify releases in matter of days.  
> It does set a limit to how much can be tested in the limited time  
> we have. We have an outstanding example just from last friday where  
> we almost missed a rather critical regression because we don't have  
> an automated test for such scenarios and even worse, don't have  
> time to write more tests. And there is only so much manual testing  
> that can happen in such shorter release cycles.
> Monthly cycles would give us slightly more room to do more  
> exhaustive testing and test development.
>
> So if we do go ahead with the weekly releases, the usual caveat  
> does hold, that is once in a while we might have to live with bugs  
> and regressions in the release. This might necessitate rolling out  
> patches on Hub, if critical.

I agree. The current burden of manual testing is preventing us from  
being to able to shorten our release cycles on many fronts.

1) There is a minimum amount of manual QA we currently do because  
we're lacking in some of our automated test coverage.
2) The extra manual test time it takes to qualify a release is based  
on the amount of changes, the more we change the more QA we do. And  
we don't have enough time to write new tests for features in the  
branches they are developed in before being merged in for the release.
3) All this extra manual testing time is preventing us from having  
good blocks of time to invest in additional automated test coverage.

IMHO the only way to shorten the release cycles is to abolish manual  
testing and restrict QA's manual testing efforts to bug regressions.

-Mikeal