[Cosmo-dev] Collaborative QA for Preview

[Ted Leung]

Hi Aparna,

Some thoughts in line...

On Feb 23, 2007, at 12:18 AM, Aparna Kadakia wrote:

> Preview release, as we all know, is a big milestone release for  
> OSAF. Obviously, our quality standards are going to be very high  
> for this release. As a 4 member QA team, divided between Cosmo and  
> Chandler, we are grossly understaffed for doing detailed,  
> exhaustive testing of all the new features. Without going on a mass  
> hiring spree, we would like to brainstorm on ideas about how we  
> could collectively as a team commit ourselves to raising the  
> quality standards for this release.
>
> Some of the things we think would help are:
> 1. More automation
> Add more automated tests to the test suite. This would especially  
> help in regression testing of existing features by reducing manual  
> cycles. This also includes adding unit tests for all the new  
> features being checked in. So when we hit Feature Complete  
> deadline, it would mean all the unit tests for the specific feature  
> are also done. These would be run continously on tinderboxes and  
> thereby help catch any regressions.

I agree that more automation will help in some areas.   I would like  
to get to a point where Cosmo developers could run windmill tests  
against their changes before those changes get checked in.   When I  
worked on Chandler, I used to run the functional tests before I  
checked in a unit of work, and it saved me from embarrassment many a  
time.

One area where automation will not help is in interop testing between  
Chandler and Cosmo.  This is an are where we could do with some more  
testing by humans.

>
> 2. More pariticipation in the IRC QA sessions
> As you all know we now have a dedicated IRC office hour for QA  
> sessions on wednesdays at 11:00 AM PST. Traditionally only the QA  
> team, PPD team and the dev managers have been participating in  
> these sessions. What we would like is to see is more participation  
> from other team members, especially in cross-product and cross- 
> feature testing. Hopefully this is not asking for too much. Just  
> like we make time for staff meetings, design sessions etc, we  
> should make time for this in our schedule and commit ourselves to  
> exercising the app in different ways and testing new features to  
> help uncover bugs.

At an hour a week, this is not a big of a time commitment.

>
> 3. More internal dogfooding
> If we could just get people internally to download Chandler  
> periodically and use it for maintaining their personal calendars as  
> well as updating the office calendar that would be a big help.

I think that we should now be in a situation where everyone can  
either use Chandler or Cosmo to update the office calendar.     I am  
maintaining my work calendar using Chandler, and I will be posting a  
Cosmo bookmarkable URL so that people can view/edit calendars.   I am  
going to stop replying to e-mail requests for my availability and  
start replying with that bookmarkable URL.

>
> The other suggestions we have heard are:
> 1. Maybe increasing the frequency of the IRC QA sessions from once  
> a week to twice a week with partipation from all prouct teams.
> 2. All the teams dedicate 1 day a week to testing after feature freeze
> 3. Hiring a contractor till preview ( I am not sure how much value  
> this would add)
> We would like to hear more suggestions if you have any that have  
> worked in your previous companies.
>
> A few things to remember that the bugs in the software will turn up  
> eventually and we will need to fix all of them before the release.  
> So it would be most efficient on everyone's time in finding them  
> earlier than later. Also when you cross test features, you bring  
> fresh perspectives on usability of the feature, uncovering any  
> design bugs earlier on, if any. This would be valuable user  
> feedback for the PPD team as well.
>
> At the end of the day we need to release quality software and if  
> that means we need to get more creative and figure out ways in  
> which we can leverage the strength of the team for getting there I  
> sure hope we can do that.

In many open source projects, the developers also spend time testing  
the software.  I don't see why we can't do the same.

Ted