[Chandler-dev] Comments requested on new test output format
mikeal at osafoundation.org
Wed Jul 12 15:40:22 PDT 2006
On Jul 12, 2006, at 11:30 AM, Brian Moseley wrote:
> On 7/12/06, Dan Steinicke <dan at osafoundation.org> wrote:
>> Test Report;
>> *Suite ""ChandlerTestSuite"" Failed :: Total Time
>> ""0:04:55.434000"" ::
>> Comment ""None""
>> **Test ""TestSwitchTimezone"" Failed :: Total Time
>> ""0:00:00.931000"" ::
>> Comment ""None
>> ***Action ""CheckBlockVisibility"" Failed :: Total Time
>> ""0:00:00"" ::
>> Comment ""(On EditTimeZone Visibility) || detail view = Fa
>> lse ; expected value = True""
>> ****Report ""(On EditTimeZone Visibility) || detail view = False ;
>> expected value = True"" Failed :: Comment ""None""
> this is way, way too much output for me. i should be able to scan the
> output and very quickly see everything that happened. i don't need
> verbose descriptions of tests, and i don't need to know anything about
> passing tests at all other than that they passed.
> i want test runs to look like this:
> 1: switching timezone... ok
> 2: checking block visibility... skipped (not enabled on this platform)
> 3: editing timezone visibility... failed (you can't do that, silly!)
> 4: something else... ok
I think there is a misunderstanding of what is actually being shown.
This is the "end of test report", which ONLY shows the failures.
The '*' character is there to expose the encapsulation, which is why
the indent increases with each line. So this test is _not_ saying
-switching timezone - works
-checking block - skipped
-editing timezone - failed
It's saying that there was a single failure. That TestSwitchTimeZone
failed. It failed in "CheckBlock Visibility" when checking
If you're suggesting that the report just say "I ran 5 test and
"TestSwitchTimeZone" " failed, and that anything more is too much
output then I'll have to disagree. The purpose of the automated
testing system is to tell you, as granularly as possible _what_
failed, not just that _something_ failed.
A given test performs between 5 and 400 individual actions. Just
telling you that it "failed somewhere" is nearly useless, all that
says is that you need to go and track some issue, somewhere. Instead
this tells you what failed, and where, and in some of the test tools
you could even run that single test again with an increased debug
level and get more feedback (like in HTTPTest it will print the
requests and responses from the server if debug > 4).
> Total tests run: 25 Passed: 23 Failed: 1 Skipped: 1
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> Open Source Applications Foundation "chandler-dev" mailing list
More information about the chandler-dev