[Chandler-dev] Comments requested on new test output format

Mikeal Rogers mikeal at osafoundation.org
Wed Jul 12 15:40:22 PDT 2006


On Jul 12, 2006, at 11:30 AM, Brian Moseley wrote:

> On 7/12/06, Dan Steinicke <dan at osafoundation.org> wrote:
>
>> Test Report;
>> *Suite ""ChandlerTestSuite"" Failed :: Total Time  
>> ""0:04:55.434000"" ::
>> Comment ""None""
>> **Test ""TestSwitchTimezone"" Failed :: Total Time  
>> ""0:00:00.931000"" ::
>> Comment ""None
>> None""
>> ***Action ""CheckBlockVisibility"" Failed :: Total Time  
>> ""0:00:00"" ::
>> Comment ""(On EditTimeZone Visibility)  || detail view = Fa
>> lse ; expected value = True""
>> ****Report ""(On EditTimeZone Visibility)  || detail view = False ;
>> expected value = True"" Failed :: Comment ""None""
>
> this is way, way too much output for me. i should be able to scan the
> output and very quickly see everything that happened. i don't need
> verbose descriptions of tests, and i don't need to know anything about
> passing tests at all other than that they passed.
>
> i want test runs to look like this:
>
> 1: switching timezone... ok
> 2: checking block visibility... skipped (not enabled on this platform)
> 3: editing timezone visibility... failed (you can't do that, silly!)
> 4: something else... ok

I think there is a misunderstanding of what is actually being shown.

This is the "end of test report", which ONLY shows the failures.

The '*' character is there to expose the encapsulation, which is why  
the indent increases with each line. So this test is _not_ saying

-switching timezone - works
-checking block - skipped
-editing timezone - failed

It's saying that there was a single failure. That TestSwitchTimeZone  
failed. It failed in "CheckBlock Visibility" when checking  
"EditTimeZone" visitibily.

If you're suggesting that the report just say "I ran 5 test and  
"TestSwitchTimeZone" " failed, and that anything more is too much  
output then I'll have to disagree. The purpose of the automated  
testing system is to tell you, as granularly as possible _what_  
failed, not just that _something_ failed.

A given test performs between 5 and 400 individual actions. Just  
telling you that it "failed somewhere" is nearly useless, all that  
says is that you need to go and track some issue, somewhere. Instead  
this tells you what failed, and where, and in some of the test tools  
you could even run that single test again with an increased debug  
level and get more feedback (like in HTTPTest it will print the  
requests and responses from the server if debug > 4).

-Mikeal

> ....
> Total tests run: 25 Passed: 23 Failed: 1 Skipped: 1
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "chandler-dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/chandler-dev



More information about the chandler-dev mailing list