[Dev] Re: Functional Tests Fail: Tinderbox reports success

[Mikeal Rogers]

In Line.

-Mikeal

On Feb 13, 2006, at 6:06 PM, John Anderson wrote:

>
>
> Aparna Kadakia wrote:
>>>> Finally, I noticed that TestNewCollection, when run alone fails by
>>>> hanging forever, but when run in the order of the functional tests,
>>>> fails with an attribute error. I was surprised to learn that each
>>>> functional test isn't run from the same known starting point,  
>>>> e.g. they
>>>> are all run one after another in the same execution of Chandler.  
>>>> For
>>>> someone like me, who just wants to add a new functional test, it  
>>>> would
>>>> be convenient if my test results didn't depend on the random  
>>>> state of
>>>> Chandler left over after the tests that ran before it --  
>>>> especially when
>>>> some of the test access network resources.
>>>>
>>
>> We haven't seen too many problems with running all the tests  
>> together in the test suite so far. More often than not it has  
>> discovered real problems. We do have the flexibility of running  
>> the tests individually on a new repo if we have to.
>> But I am open to the idea of having a functional test suite that  
>> runs each test on a new repo copied over if that helps the  
>> developers debugging their tests.
>>
> After looking into the TestNewCollection failure, it looks like it  
> is a real bug, so I now see the benefit of running them together.  
> How about if we run them both separately and together? That way  
> we'll know if it is an order related bug.

Grant had a suggestion today that it may be nice to have an option  
that restarted chandler and cleared the repository in between tests,  
so each test in a suite ran in a clean state.

This would be a slightly cleaner way of running them separately than  
stringing together a bunch of do_test commands and we could even  
capture the log output and get PASS or FAIL for the entire suite. But  
this would require some work be done on do_tests.

This feature is simple in the new framework arch but there is no  
clear delivery date for that, so if we want this feature in the short  
term it will need to be done in do_test or as a short term fix we  
would have to string together separate do_test runs.

-Mikeal

>
> John
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/dev