[Cosmo-dev] Cosmo release frequency

[Mikeal Rogers]

Jared's comments made me do a little thinking.

The update every day scenario in particular has me tickled. I've been  
thinking of what it might take to make that happen, or for that  
matter weekly of bi-weekly releases.

In my opinion doing bi-weekly releases the way we've done these last  
4 weekly 0.7.x releases is unacceptable for everyone. The main  
problem is that our automated coverage is supplemented with manual  
testing, and we've focused that manual testing on areas of  
functionality we know have changed. If too much changes the burden of  
manual testing is too great and the QA time for that release would  
increase enough to hamper us from having enough time to improve our  
automated test coverage over time.

In terms of the WebUI, there isn't anything we can't automate the  
testing of. Everything is "testable" (except SSL, fixing in windmill  
0.3). It's just a matter of freeing up enough of our time to write it  
all, and to write tests against builds that may be in branches where  
a feature is being completed before release.

The backend is where things get a big tricky. We're constantly trying  
to broaden the amount of clients we interop with, and doing that  
interop testing is all manual. We have some backend protocol tests  
but if you look at the traffic from just two of the clients we  
interop with you'll see that handling enough cases using our backend  
automated protocol tests would be very exhausting and we'd still end  
up not covering some of the edge cases that manual client interop finds.

<begin sidetrack>
I'd like to take a moment and say that I'm very impressed with the  
amount of testing the development team has taken on itself in this  
last release. The backend team seems to have been doing interop  
testing with other clients as they go and fixing issues as they go.  
Much much kudos.
<end sidetrack>

In my mind this doesn't mean we can't do shorter releases, we just  
have to change our process and what we expect from these releases and  
from QA.

The major barriers in my opinion are matters of policy and perception  
and the technical/practical hurdles are easily overcome. Currently  
our policy is to create very strong and seemingly bug free releases,  
and the perception is that we end up with fewer bugs and a better  
experience for our users. The truth is that no release is truly bug  
free, and in most cases the "blocking" issues we find are exaggerated  
because it just takes so damn long to role up another release ( even  
a week is too long for many issues), and are usually found late  
because QA is tied up in release testing more than it is writing new  
test cases.

If all of the release testing was automated, and the manual  
interactions minimized to Adam just watching the tests as they ran to  
make sure no obvious extra pieces of UI are being drawn then we could  
release every other day.

If we decided that after 0.8 the only manual QA done on a release for  
interop was regressing the interop bugs that were fixed, we could  
release every other day.

What this means is that we are stating unequivocally that we are  
shipping releases with bugs. We'll have some regressions, and we'll  
have some bugs go under the radar, but once a user runs in to these  
problems we can release a fix in either hours or days depending on  
the urgency of the bug. The effect of this change in process would  
mean that the practical impact of a bug is much much lower and with  
QA freed from manual testing we would have a lot more time to improve  
the automated test coverage and each release would have less of a  
chance of regressions than the previous.

There is a long list of additionally infrastructure that QA would  
need to make this happen, some of it has been covered in previous  
mails and some has not. If we start heading towards a consensus that  
this is route we want to take in our process then I will outline them  
in another reply to this thread.

I'd also like to mention that whether every other day or every other  
week the issues for QA IMHO are pretty much the same. I can't think  
of any extra issues we would have with releasing every other day that  
we wouldn't have every other week, we would really need all the same  
changes in process and infrastructure.

-Mikeal

On Oct 8, 2007, at 2:21 PM, Jared Rhine wrote:

> Ted Leung wrote:
>> I'd like to go back to a time based release schedule, but I'd like  
>> to have people's opinions as to how frequently we ought to release.
>
> To avoid a treatise on software development philosophy, I'll  
> restrict my comments to just "Hub stakeholder".
>
> Short release cycles are good for the Hub.
>
> Short cycles minimize the amount of stuff that breaks at once.   
> Short cycles also minimize look-and-feel changes so that regular  
> users aren't shocked as frequently.
>
> They get features into users' hands quicker, which helps everyone  
> guide ("steer") subsequent development.
>
> Ultimately, long-term Hub success is based more on features  
> delivered in total; whether those features are spread of 1, 2, 4,  
> or 30 releases per month actually doesn't matter that much.  It  
> seems like there's consensus at least at >1 month releases are too  
> long to "steer" properly.
>
> I'm hearing from others that they think we'd deliver more total  
> features on a monthly cycle than on a biweekly cycle, though that's  
> contrary to my personal belief and experience.
>
> I think we see a fair amount of inter-and-intragroup stalling on  
> longer cycles, in particular when we need a number of RCs before  
> final release.  This is my primary negative perception about month- 
> long cycles.
>
> If it was just about the service, I would want to look seriously  
> into a real Agile deployment, see Flickr's "release every 30 minutes":
>
>   http://blogs.warwick.ac.uk/chrismay/tag/flickr/
>
> Related ideas include "update the Hub after each tested trunk  
> checkin", "update daily", etc.  There's much in our existing  
> processes and beliefs that makes such ideas a non-starter  
> currently, but I'd push back on people that say it's not possible  
> or realistic.  The blockers to such a route seem more philosophical  
> than technical (ala, what must be QA'ed to be doing QA at all, what  
> does it mean to produce a software release, etc.)
>
> I don't want to oversell this Hub-centric view to the Cosmo team.   
> The final decision will be a balance of considerations and it's  
> really important that the Cosmo team is happy with Cosmo  
> development.  The Hub view seems to run counter to developer's  
> stated views, so the owner/driver will figure out how to balance  
> those.
>
> Travis and I discussed over lunch some other crazier ideas.  A  
> separate front-end and back-end project would allow the front-end  
> to update on a faster cycle than the backend.  It would also allow  
> the simulataneous production hosting of a hub.chandlerproject.org  
> and beta.hub.chandlerproject.org instances pointed at the same  
> Cosmo backend.
>
> -- Jared
> _______________________________________________
> cosmo-dev mailing list
> cosmo-dev at lists.osafoundation.org
> http://lists.osafoundation.org/mailman/listinfo/cosmo-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osafoundation.org/pipermail/cosmo-dev/attachments/20071008/7d0695fc/attachment.html