[Chandler-dev] Comments requested on new test output format

Dan Steinicke dan at osafoundation.org
Wed Jul 12 13:39:35 PDT 2006


Hi Brian,

I personally went through the commit logs for the changes to the tests 
in the old framework and updated the
new tests with these same changes.  Your i18n changes were the bulk of 
this (including adding the uw() method)
but there were also a number of changes from other devs.  The amount of 
effort that must now be expended to maintain
two sets of tests is the main reason I am now pushing for moving quickly 
forward to using the new tests.  Once
the new tests are running on tinderbox without problems the plan is to 
stop maintaining the old tests.

The new framework is mainly about how the tests are run and logged, with 
little change to the function of the tests themselves.
 
I don't believe we have made any special efforts to adapt the CATS-0.2 
framework for i18n testing specifically, other than the
new framework being more flexible and easier to develop tests for.  My 
first thought on this is that we should
continue to push ahead to get the new framework working on tinderbox and 
once that is accomplished look work on the
modifications you mention to test for i18n issues.  I would welcome your 
input on the new framework and its suitability to
testing i18n issues and expect to work closely with you to develop this 
functionality in the future.  (links to docs in original msg)

If you wish to continue this discussion, can we start a new thread?  
This an important issue but one that is pretty far afield from
my original request seeking comment on the new logger output.

Dan

Brian Kirsch wrote:
> Hi Dan,
> It is very important that CATS-0.2 be design with internationalization 
> support in mind.
> The most common issues / bugs that arise related to i18n are subtle 
> and a good testing
> framework is essential.
>
> Could you detail what work / thought has gone in to CATS-0.2 regarding 
> i18n and what is left to be done.
>
> Specifically in CATS-0.1 I went through and wrapped all displayable 
> strings in uw() methods from the
> i18n.tests package. The uw method inserted a semi-random unicode 
> character at the beginning of the
> displayable string and a unicode character at the end of the 
> displayable string.
>
> Adding this uw() wrap was key to finding many subtle bugs including 
> incorrectly logging
> unicode characters and not encoding unicode to bytes when using the 
> Python file system API's.
>
> It even uncovered issues with the CATS-0.1 framework itself including 
> a few places where items were
> converted to strings i.e. str(item) which is a no no!
>
> Using unicode strings is only a small aspect of an i18n testing 
> framework.
>
> In CATS I would like the framework to provide support for running in 
> different locales and of course timezones.
>
> For each locale the expected output for many operations will change. 
> For example a date / time string will
> be different, the first day or the week in a calendar can be 
> different,  The UI menu labels and auto generated
> collection names will be different.
>
> I am happy to sync up with you guys but I do want to make sure at this 
> stage we are writing frameworks that
> meet the needs of Chandler.
>
> Almost all of the Internationalization work will be completed in the 
> Alpha 4 time frame.
>
> The testing framework is the key to preventing regression bugs in i18n 
> as we drive towards Chandler 1.0.
>
> Thanks,
> Brian
>
>
> Dan Steinicke wrote:
>
>> We are getting close to being able to switch over to the new test 
>> framework (CATS-0.2).  As it is currently implemented the output from 
>> the tests will be quite different from the current test output.
>> Please take a moment to look over the test output samples below and 
>> let us know what you think so we can address peoples concerns before 
>> the new framework goes live, rather than after.
>>
>> Two sample outputs are shown below, one where a test fails without a 
>> traceback, another of a failure with a traceback.
>>
>> For more info on the new framework see :
>> general docs 
>> http://wiki.osafoundation.org/bin/view/Projects/ChandlerAutomatedTestSystemZeroPointTwo 
>>
>> writing new tests 
>> http://wiki.osafoundation.org/bin/view/Projects/WritingChandlerAutomatedTestsWithCATSZeroPointTwo 
>>
>>
>> Thanks
>> Dan
>>
>> ############# Here is sample output from do_tests where one tests 
>> fails without a traceback:
>>
>> Test Report;
>> *Suite ""ChandlerTestSuite"" Failed :: Total Time ""0:04:55.434000"" 
>> :: Comment ""None""
>> **Test ""TestSwitchTimezone"" Failed :: Total Time ""0:00:00.931000"" 
>> :: Comment ""None
>> None""
>> ***Action ""CheckBlockVisibility"" Failed :: Total Time ""0:00:00"" 
>> :: Comment ""(On EditTimeZone Visibility)  || detail view = Fa
>> lse ; expected value = True""
>> ****Report ""(On EditTimeZone Visibility)  || detail view = False ; 
>> expected value = True"" Failed :: Comment ""None""
>> ***Action ""CheckEditableBlock"" Failed :: Total Time ""0:00:00"" :: 
>> Comment ""(On EditTimeZone Checking)  || detail view value =
>> Floating ; expected value = US/Pacific""
>> ****Report ""(On EditTimeZone Checking)  || detail view value = 
>> Floating ; expected value = US/Pacific"" Failed :: Comment ""None"
>> "
>> ***Action ""CheckBlockVisibility"" Failed :: Total Time ""0:00:00"" 
>> :: Comment ""(On EditTimeZone Visibility)  || detail view = Fa
>> lse ; expected value = True""
>> ****Report ""(On EditTimeZone Visibility)  || detail view = False ; 
>> expected value = True"" Failed :: Comment ""None""
>> ***Action ""CheckEditableBlock"" Failed :: Total Time ""0:00:00"" :: 
>> Comment ""(On EditTimeZone Checking)  || detail view value =
>> Floating ; expected value = US/Pacific""
>> ****Report ""(On EditTimeZone Checking)  || detail view value = 
>> Floating ; expected value = US/Pacific"" Failed :: Comment ""None"
>> "
>> $Suites run=1, pass=0, fail=1 :: Tests run=26, pass=25, fail=1 :: 
>> Actions run=367, pass=363, fail=4 :: Reports run=556, pass=552,
>> fail=4
>> #TINDERBOX# Testname = ChandlerTestSuite
>> #TINDERBOX# Time elapsed = 0:04:55.434000 (seconds)
>> #TINDERBOX# Status = FAILED
>> - + - + - + - + - + - + - + - + - + - + - + - + - + - + - +
>> The following tests failed
>>   
>> (debug)C:\cygwin\home\Dan\chandler\tools\cats\Functional\FunctionalTestSuite.py 
>>
>>   
>> (release)C:\cygwin\home\Dan\chandler\tools\cats\Functional\FunctionalTestSuite.py 
>>
>>
>>
>> ########## here is some sample output of a test failing with a traceback
>>
>>
>> Test Report;
>> *Suite ""ChandlerTestSuite"" Failed :: Total Time ""0:00:06.139000"" 
>> :: Comment ""None""
>> **Test ""TestCauseTrace"" Failed :: Total Time ""0:00:00.040000"" :: 
>> Comment ""None
>> Test Failure due to traceback
>> Traceback (most recent call last):
>> File "C:\cygwin\home\Dan\chandler\tools\cats\framework\runTests.py", 
>> line 51, in run_tests
>>   test.runTest()
>> File 
>> "C:\cygwin\home\Dan\chandler\tools\cats\framework\ChandlerTestCase.py", 
>> line 68, in runTest
>>   self.startTest()
>> File 
>> "C:\cygwin\home\Dan\chandler\tools\cats\Functional\TestCauseTrace.py", 
>> line 24, in startTest
>>   1/0
>> ZeroDivisionError: integer division or modulo by zero
>> ""
>> ***Action ""Divide by zero "" Failed :: Total Time ""0:00:00.040000"" 
>> :: Comment ""None""
>> ****Report ""Action Failure due to traceback"" Failed :: Comment 
>> ""None""
>> $Suites run=1, pass=0, fail=1 :: Tests run=3, pass=2, fail=1 :: 
>> Actions run=5, pass=4, fail=1 :: Reports run=28, pass=27, fail=1
>> #TINDERBOX# Testname = ChandlerTestSuite
>> #TINDERBOX# Time elapsed = 0:00:06.139000 (seconds)
>> #TINDERBOX# Status = FAILED
>>
>> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>>
>> Open Source Applications Foundation "chandler-dev" mailing list
>> http://lists.osafoundation.org/mailman/listinfo/chandler-dev
>
>


More information about the chandler-dev mailing list