[Chandler-dev] Comments requested on new test output format
mikeal at osafoundation.org
Wed Jul 12 17:44:21 PDT 2006
On Jul 12, 2006, at 5:19 PM, Brian Moseley wrote:
> On 7/12/06, Mikeal Rogers <mikeal at osafoundation.org> wrote:
>> This output;
>> is only displayed when there is a failure. It's just displayed before
>> the final summary. It does however "spew" to your screen every time
>> there are failures calculated at the end of the suite.
> it's too verbose. all the "" and :: and * and ** and Comment and so
> forth. just using intelligent whitespacing. the other stuff just makes
> me glaze over immediately.
>> I'm 50% done writing the new framework (OAF). With OAF you can
>> specify totally different output to the file than to stdout.
> i want summary output at the console and detailed trace info in a log.
> can it do that?
Yup, you can even have the format look completely different. Like if
you wanted easily parsable output in console and readable output in
You could even have two log files, and the information in each is
formatted differently and masks more or less. You can even add
another output object that inserts rows into a database, or sticks a
file on a webdav server for every line of output.... the
possibilities are ENDLESS :)
>> The 'trace' is what we are trying to show with the encapsulation
>> using the '*' character, maybe we're doing a bad job of it and should
>> consider a different format for the output (Have something that looks
>> more like Python tracebacks?), but the content we are showing is all
>> meaningful to the failure.
> i'd rather not see this in my console. i'd rather grep through a
> log file.
For now, it's all or nothing in CATS-0.2. CATS-0.3 and broadsword-0.2
will have this option, and can be easily tailored to any use case.
>> The summary you describe is printed at the end with (Number of Test
>> Ran, Number of Test Failures, etc), we can easily add in a few lines
>> that say "these are the tests that failed'. In this case you could
>> just ignore any output until the last few lines.
> i don't want the test tool filling 800 lines of scrollback buffer if
> i'm only going to look at the final 3 lines. just show me a list of
> names of tests that failed so that i can find them in the log to look
> at their stack traces.
Agreed. But.... for now it's all or nothing in CATS-0.2. If you want
more output you'd have to run it again with some debugging attributes
>> So are you saying that the output is too much, or that it's not
>> readable enough, or both? The content in the failure report is all
>> necessary to find the point of failure.
> both, and i disagree. the time elapsed for the test suite and failed
> tests are not needed to understand what failed. only the failing
> test's names.
Ok, the time elapsed isn't necessary in finding most bugs, point
taken. But the failed test -> action -> fail report is.
For example we have one generic function that checks a bunch of
fields in Chandler, we run it multiple times during each test. The
only different between the first time we run it and the second is
what action it was in, which is why we have the trace.
Now the detailed information about the test -> action is probably
unnecessary for purposes of the trace, just the names and comments if
any exist would be good.
But if you have a different view and want less output then you'll be
able to do that, just not in this release of CATS.
> when i look at my console, i want to see that every test has been run
> and which ones have failed. my original suggestion achieves this
> admirably. fwiw, it is the standard format (maybe with a tweak or two)
> for perl's test harness which has been in use for ten plus years.
Point taken. But I don't think there is one solution that is going to
work for everyone for this problem, different people will want things
to look differently and have differing expectations.
OAF can be customized to fit anyones needs, for now we just need to
tweak this output as to offend people the least until we finish OAF
>> Concealed all output during the test run except
>> Starting TestBlah
>> Ending TestBlah
>> Starting Test2Blah
>> ----Something failed
>> Ending Test2Blah
> that is way too much information. how about:
> TestBlah ... ok
> Test2Blah ... failed ("this is the reason the test failed")
We can't display this until the suite has ended. The way the output
object currently holds the data is such that it doesn't know which
upper level it's failed until the end of the suite when it parses the
data structure and bubbles up the results.
We can make the end of suite summary look like this in CATS-0.2. In
CATS-0.3 we can have this kind of output in real time.
> see how much more compact that is, and how it communicates exactly the
> same info in 2 lines instead of 5, and how it's extremely easy to pick
> out the failure when scanning the console's scrollback?
The encapsulation runs a bit deeper than Suite runs Tests and tests
fail for some reason. It's more granular, Tests contain actions which
have many pass/fail reports in them, which is what we we're trying to
show with the traceback. If you don't care about that until you look
in the log file then that's fine, we can accomplish this in OAF and
give you the output you want.
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Open Source Applications Foundation "chandler-dev" mailing list
More information about the chandler-dev