[cosmo-dev] Pre-RC automation tests

[Ted Leung]

On Nov 28, 2007, at 12:45 PM, Mikeal Rogers wrote:

>
> On Nov 28, 2007, at November 28, 200712:13 PM, Brian Moseley wrote:
>
>> On Nov 28, 2007 11:22 AM, Jared Rhine <jared at wordzoo.com> wrote:
>>
>>> In short: Should protocol tests be run before RCs are declared?
>>
>> yes.
>
> This is a no brainer, these tests run in under a minute. The only  
> blocker here now is that next.osaf.us is a migrated instance so you  
> need an admin account to pass to the protocol tests so that they  
> can create new accounts and do cmp tests.
>
>>
>>
>>> How about migration tests?
>>
>> if migration code changed since the previous rc, then yes.
>
> These should be _run_, but it takes some time to get feedback from  
> the interested parties looking at the log. This isn't a pass or  
> fail kind of test.

Based on the last few releases - 0.8/0.8.1 and 0.9/0.9.1  it seems  
like we haven't yet reached the point where our automation is able to  
catch showstopping brown paper bag bugs.    I think that we'd be well  
served to have an impromptu collaborative QA session at some point in  
the process, either in the final pre-RC stage that we are discussing  
now, or for the first RC.

Ted