[Dev] Re: Functional Tests Fail: Tinderbox reports success
john at osafoundation.org
Mon Feb 13 18:06:35 PST 2006
Aparna Kadakia wrote:
>>> Finally, I noticed that TestNewCollection, when run alone fails by
>>> hanging forever, but when run in the order of the functional tests,
>>> fails with an attribute error. I was surprised to learn that each
>>> functional test isn't run from the same known starting point, e.g. they
>>> are all run one after another in the same execution of Chandler. For
>>> someone like me, who just wants to add a new functional test, it would
>>> be convenient if my test results didn't depend on the random state of
>>> Chandler left over after the tests that ran before it -- especially
>>> some of the test access network resources.
> We haven't seen too many problems with running all the tests together
> in the test suite so far. More often than not it has discovered real
> problems. We do have the flexibility of running the tests individually
> on a new repo if we have to.
> But I am open to the idea of having a functional test suite that runs
> each test on a new repo copied over if that helps the developers
> debugging their tests.
After looking into the TestNewCollection failure, it looks like it is a
real bug, so I now see the benefit of running them together. How about
if we run them both separately and together? That way we'll know if it
is an order related bug.
More information about the Dev