[cosmo-dev] Pre-RC automation tests

[Aparna Kadakia]

>
> Based on the last few releases - 0.8/0.8.1 and 0.9/0.9.1  it seems  
> like we haven't yet reached the point where our automation is able  
> to catch showstopping brown paper bag bugs.    I think that we'd be  
> well served to have an impromptu collaborative QA session at some  
> point in the process, either in the final pre-RC stage that we are  
> discussing now, or for the first RC.
>
I believe we have been doing collaborative migration test session or  
every release that has significant data migration involved. And we  
should continue doing that. Having said that,  I don't believe those  
sessions have really helped us catch any of "brown paper bag" bugs  
(for those wanting to know what is a brown paper bag bug see below )  
or atleast not that I recall.
We don't land up finding most of these crafty bugs up until people  
start using the app for their daily workflows.
As for doing the QA sessions, I vote for doing them once an RC is  
done. Migration testing is sort of open ended and we can't stall  
creating an RC till all of migration testing is completely.


~
what is a brown paper bag bug:
  A bug in a public software release that is so embarrassing that the  
author notionally wears a brown paper bag over his head for a while  
so he won't be recognized on the net. Entered popular usage after the  
early-1999 release of the first Linux 2.2, which had one. The phrase  
was used in Linus Torvalds's apology posting.