[Chandler-dev] Comments requested on new test output format
bcm at osafoundation.org
Wed Jul 12 15:54:47 PDT 2006
On 7/12/06, Mikeal Rogers <mikeal at osafoundation.org> wrote:
> If you're suggesting that the report just say "I ran 5 test and
> "TestSwitchTimeZone" " failed, and that anything more is too much
> output then I'll have to disagree. The purpose of the automated
> testing system is to tell you, as granularly as possible _what_
> failed, not just that _something_ failed.
that's what i'm saying. there's a difference between detailed trace
info and a simple summary of test results. 99% of the time i want to
only see the simple results. when there's a failure, i'm happy to go
look in a log file for the trace info. i definitely don't want it all
spewed to my screen for every test run.
> A given test performs between 5 and 400 individual actions. Just
> telling you that it "failed somewhere" is nearly useless, all that
what i meant to communicate was that, along with failure, the test
system would report a meaningful message (provided by the test author)
describing what went wrong and providing a starting point for
investigation. detailed trace info in a log file would help narrow the
> says is that you need to go and track some issue, somewhere. Instead
> this tells you what failed, and where, and in some of the test tools
> you could even run that single test again with an increased debug
> level and get more feedback (like in HTTPTest it will print the
> requests and responses from the server if debug > 4).
in the normal case i want as little feedback as possible, and i want
it to be supremely easy to see when something failed and what specific
thing(s) failed. that is just not the case with HTTPTest or with the
suggested output from earlier in this thread.
More information about the chandler-dev