[Cosmo-dev] Cosmo QA process alterations

[Aparna Kadakia]

Team

Having been through 4 (will be 5 today) back to back Cosmo releases,  
we have discovered some inefficiencies in the process that we would  
like to rectify in the upcoming releases. Given the short release  
cycles in cosmo, QA has been made available for no time to develop  
more automated functional tests. As we all know, having more  
automated tests is the single most critical factor for the success of  
shorter releases.

The largest barrier to a faster QA release cycle ( faster being less  
than 1 day to certify a release ) we need to remove the block of time  
we current have alloted for manual functional testing, almost all of  
which _can_ be automated but hasn't been done yet. Since the recent  
releases allowed us no time to improve this area of automated testing  
we can assume that the upcoming two week release cycles which will  
include larger new features and won't allow us any time either. Since  
going forward we will be automating new features as they come in to  
the product we may even start lagging further behind in automated  
coverage and picking up the slack even more with manual testing.

Practically, the only automated testing we can't automate is bug  
regressions and interop. Everything else can be automated if QA's  
time isn't locked up in manual release testing.

Since we have already agreed on the fact that we are going to  
experiment with 2 week releases for Cosmo going forward, to solve  
this is problem in a meaningful way without causing any significant  
changes to the schedule or other release processes we propose the  
following:

1. For the next 2 releases, QA would like to engage in no manual  
testing beyond bug regressions and interop, and instead devote all  
their time to developing additional automated tests. These tests  
would be a replacement for the manual tests that are currently being  
done, and a more diligent process to automate any new bugs found.  
Only the automated Windmill and protocol tests will be run, resolved  
bugs will be verified manually, and accceptance level interop testing  
will be done to certify a release.

2. The down side to this would be that there is a chance the next 2  
releases may be buggy and we may need to dish out patch releases in  
case any critical bugs are encountered. But given this new process QA  
can certify a new patch release in a matter of hours. The advantage  
in the long term is we will have far broader and greater quality  
coverage with automated tests. Currently Adam's manual testing has  
helped catch less than a dozen bugs in all of the 0.7.x releases.  So  
we are not really looking at widely exposing ourselves by not running  
the manual tests for the next 2 releases.

3. Another change in the process we would like is for QA to work on  
automating new features in the branch they are developed in before  
that feature is dropped in to the trunk. The next 2 releases may  
potentially have big features like calendar overlays, month view and  
date picker. We would much rather devote the time to writing more  
automated tests for these features upfront rather than playing catch  
up in following releases or during the release certification and  
resorting to manual testing to cover what we couldn't squeeze in.

4. Ideally developers can write very basic workflow testcases for UI  
features along with their unittetss before QA steps in to cover the  
more intricate working scenarios. This will ensure that QA has  
everything it needs and is not blocked in any way when taking on a  
feature to automate. This will all insure a higher quality of the  
feature when it lands in trunk.

Questions, concerns?

Let us know,
Aparna