[cosmo-dev] Release terminology

[Mikeal Rogers]

Recently we've had some cross group communication problems.

It seems that over the last few releases people have had a different  
understanding of what each group ( particularly QA ) is doing at  
various points in the release. I think the biggest problem here is  
that we've made huge changes to our process for release, handoff, and  
testing, but we haven't modified any of the terminology. Even worse,  
our process differs from the Desktop but still using many of the same  
terms to describe the stages of our process.

This lead to the beginning of a thread ""what is pre- post- RC?"" but  
after talking with Ted, Aparna, and Jared I realized the problem is a  
little larger than that.

= What our release looks like now =

Development works on a series of features in branches, when those  
features are completed they get merged into the trunk for the next  
release. Known issues and bugs are fixed in the new features and once  
there are zero open bugs QA begins the release testing process.

The complication here is that QA may have been working in those  
branches writing new tests for features as they were being developed.  
Or QA may have been too busy, usually working with a previous release  
stretching on a bit longer than planned, and may not have had time to  
write new tests for those features as they were being developed in the  
branch.

This means that after the "zero bug" handoff to QA for release  
testing, QA's certification time ranges between 4 or 5 hours to 4 or 5  
days. This release certification phase used to be referred to as the  
"RC" phase but that terminology is riddled with problems, it implies  
that we're done feature testing, it means an RC has been cut and  
assumes that we are testing the release build bits ( we aren't, we're  
testing builds on on lab.osaf.us that are building the same code in  
the same branch but not using the same build steps ). And until  
recently we were even trying to cut release candidates every time we  
fixed a bug -- which is cumbersome when we're actually still doing  
what we would consider "feature testing" or more accurately "new  
feature automation".

= New terminology for new process =

Ok, I'm proposing we throw out the term "RC" when describing a phase  
in the release process. We'll obvious still have "RC" builds, but I'll  
get to those later.

1 - "Open Trunk" -- This is the point at which developers begin  
merging all their branch changes in to the trunk for the next release.
1.1 It's conceivable that we branch at some point before the next  
phase. If too many developers are waiting for the trunk to open while  
bugs are getting fixed the in current release we've been known to do  
this in the past, so I would like to avoid attached the branching task  
directly to either phase.
2. "Triple Zero" -- 0 open untargetted bugs, 0 open bugs in the  
release, 0 automated test failures
    - If the release branch hasn't happened yet, it should happen now.
    - If QA wasn't automating features tests before this, they are now
    - At this point bear should do a release build and mark it RC1 so  
that outside contributors may start testing the build at home. This  
will also flush out any release build related issues.
    - QA provides an estimate to when they will be done adding new  
automated tests to this release. This timeline will obviously reflect  
how much time QA was spending on these features beforehand.
3. "Certification" -- QA is done adding new automated tests and done  
regressing bugs.
    - Obviously there are no open bugs or failing automated tests  
before this is called.
    - QA does one last automated test run through
    - Bear builds another RC build which QA sanity tests.
    - QA runs the migration script and sends it to Jared and Randy for  
review.

If it's been 24 hours since QA announced they were beginning  
"Certification" something is WRONG. No new automated tests are added,  
and certification is a 90% automated step. It should be able to be  
completed in a mater of hours. The longest step is probably just  
waiting for the migration test script to run and going through it,  
which to date hasn't taken more than 4 hours.

It's important to note who is responsible for triggering the beginning  
of each phase. Whoever is the product manager role is responsible for  
notifying the group and following up with any related task but the  
signal for them to start scrambling belongs to different players in  
each phase. "Open Trunk" this would be Ted's call. "Triple Zero"  
again, this would be development's call since QA may or may not be  
working on the release yet. And "Certification" belong to QA, bother  
triggering it and signing off.

Now it's time to open this up for debate, I'm sure I didn't get  
everything right and probably ruffled a few feathers.

-Mikeal



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/cosmo-dev/attachments/20071207/35e9b8d4/attachment.htm