[Chandler-dev] Extra check()s in functional tests
mikeal at osafoundation.org
Wed Jul 5 16:27:13 PDT 2006
On Jun 30, 2006, at 9:17 AM, Bryan Stearns wrote:
> Heikki (& Andi),
> At lunch yesterday, you (Heikki) mentioned that you'd seen a case
> where a repository problem happened after the functional test suite
> had been run (and reported passed all the tests), but failed while
> closing the repository.
> I suggested adding a test at the end that did check() on the
> repository, and you (or someone else?) suggested doing the check
> after every test script.
> I modified run_tests (called to loop over all the individual test
> scripts run by FunctionalTestSuite.py) to try this out to see what
> the time impact would be; on linuxdev1, adding one check() at the
> end added two and a half seconds, but adding a check after every
> script added almost two minutes.
> I'm sensitive to how long the tests take -- I generally have to run
> all the functional & unit tests three or four times for each
> checkin, and sometimes more -- so I don't want to add this overhead
> unless it's useful.
We have the notion of a "debug level" in CATS 0.2, although by
default we always use the same debug level we could add another
option to RunChandler that will accept the debug level, then only run
this repository test if the debug level is set greater than a certain
> Heikki, can you explain to Andi what the problem case was?
> Andi, do you think there's any utility to running after each test?
> (Seems like running once at the end would tell us if any test
> corrupted the repository; doing it for each test wouldn't add
> anything, since once it's corrupt, it's likely to stay that way.)
In the new framework this is really simple, since individual tests
run by a single method in the framework.
In the old tests scripts this would require us adding a line to the
end of every single test script. Since we are a few days away from
being at 0 bugs in CATS-0.2 and want to begin the transition after
the release of alpha3 to CATS-0.2 I think we should just wait on
adding this to each test script until we move to CATS-0.2.
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Open Source Applications Foundation "chandler-dev" mailing list
More information about the chandler-dev