[Chandler-dev] Extra check()s in functional tests

[Mikeal Rogers]

On Jun 30, 2006, at 9:17 AM, Bryan Stearns wrote:

> Heikki (& Andi),
>
> At lunch yesterday, you (Heikki) mentioned that you'd seen a case  
> where a repository problem happened after the functional test suite  
> had been run (and reported passed all the tests), but failed while  
> closing the repository.
>
> I suggested adding a test at the end that did check() on the  
> repository, and you (or someone else?) suggested doing the check  
> after every test script.
>
> I modified run_tests (called to loop over all the individual test  
> scripts run by FunctionalTestSuite.py) to try this out to see what  
> the time impact would be; on linuxdev1, adding one check() at the  
> end added two and a half seconds, but adding a check after every  
> script added almost two minutes.
>
> I'm sensitive to how long the tests take -- I generally have to run  
> all the functional & unit tests three or four times for each  
> checkin, and sometimes more -- so I don't want to add this overhead  
> unless it's useful.

We have the notion of a "debug level" in CATS 0.2, although by  
default we always use the same debug level we could add another  
option to RunChandler that will accept the debug level, then only run  
this repository test if the debug level is set greater than a certain  
number.

>
> Heikki, can you explain to Andi what the problem case was?
>
> Andi, do you think there's any utility to running after each test?  
> (Seems like running once at the end would tell us if any test  
> corrupted the repository; doing it for each test wouldn't add  
> anything, since once it's corrupt, it's likely to stay that way.)

In the new framework this is really simple, since individual tests  
run by a single method in the framework.

In the old tests scripts this would require us adding a line to the  
end of every single test script. Since we are a few days away from  
being at 0 bugs in CATS-0.2 and want to begin the transition after  
the release of alpha3 to CATS-0.2 I think we should just wait on  
adding this to each test script until we move to CATS-0.2.

>
> ...Bryan
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev