[Chandler-dev] Comments requested on new test output format

[Brian Moseley]

On 7/12/06, Mikeal Rogers <mikeal at osafoundation.org> wrote:

> This output;

> is only displayed when there is a failure. It's just displayed before
> the final summary. It does however "spew" to your screen every time
> there are failures calculated at the end of the suite.

it's too verbose. all the "" and :: and * and ** and Comment and so
forth. just using intelligent whitespacing. the other stuff just makes
me glaze over immediately.

> I'm 50% done writing the new framework (OAF). With OAF you can
> specify totally different output to the file than to stdout.

i want summary output at the console and detailed trace info in a log.
can it do that?

> The 'trace' is what we are trying to show with the encapsulation
> using the '*' character, maybe we're doing a bad job of it and should
> consider a different format for the output (Have something that looks
> more like Python tracebacks?), but the content we are showing is all
> meaningful to the failure.

i'd rather not see this in my console. i'd rather grep through a log file.

> The summary you describe is printed at the end with (Number of Test
> Ran, Number of Test Failures, etc), we can easily add in a few lines
> that say "these are the tests that failed'. In this case you could
> just ignore any output until the last few lines.

i don't want the test tool filling 800 lines of scrollback buffer if
i'm only going to look at the final 3 lines. just show me a list of
names of tests that failed so that i can find them in the log to look
at their stack traces.

> So are you saying that the output is too much, or that it's not
> readable enough, or both? The content in the failure report is all
> necessary to find the point of failure.

both, and i disagree. the time elapsed for the test suite and failed
tests are not needed to understand what failed. only the failing
test's names.

when i look at my console, i want to see that every test has been run
and which ones have failed. my original suggestion achieves this
admirably. fwiw, it is the standard format (maybe with a tweak or two)
for perl's test harness which has been in use for ten plus years.

> Concealed all output during the test run except
>
> Starting TestBlah
> Ending TestBlah
> Starting Test2Blah
> ----Something failed
> Ending Test2Blah

that is way too much information. how about:

TestBlah ... ok
Test2Blah ... failed ("this is the reason the test failed")

see how much more compact that is, and how it communicates exactly the
same info in 2 lines instead of 5, and how it's extremely easy to pick
out the failure when scanning the console's scrollback?