[Dev] Unit test runner

[Ted Leung]

Ok,

So I'm late with this commentary (to the point that the test runner, 
etc are already checked in).  So far I like what I see.

My only other comment is that I agree that we should be adding test 
methods in preference to test modules.

Ted

On Jan 26, 2005, at 1:03 PM, Phillip J. Eby wrote:

> Hi folks.  Lisa has asked me to put together a unit test runner that 
> will allow selective running of multiple unit tests, for example to 
> run just fast unit tests, or to run all tests including functional 
> tests.  I have a couple of different ways to do that, but I'd like to 
> get some input from the team as to what they'd like to see in such a 
> runner.  But before I get to the questions, I need to lay out some 
> background on Chandler's current test setup, and the options that are 
> available for how to proceed.
>
> Currently, the way most of Chandler's tests are structured, you can 
> run individual test modules with either hardhat or unittest.py, e.g.:
>
>     unittest.py repository.tests.TestText
>
> will run that specific test module, or:
>
>     unittest.py repository.tests.TestText.TestText.testBZ2Compressed
>
> to run an individual test within the module.  You can also use hardhat 
> to collect all test modules named 'Test*.py'.
>
> So, I gather that what Lisa's asking for is the ability to also run a 
> selected subset of the total collection of tests, such as just the 
> "strict" unit tests that test subsystems in isolation, with individual 
> runtimes of about 10ms or less.  (Most of Chandler's current tests are 
> technically speaking functional or integration tests, as they do not 
> test subsystems in isolation from each other.  These types of tests 
> are important also, but because they take longer to run, it's useful 
> to also have more "isolationist" tests for use during development.)
>
> Python's unittest module has a standard way of representing a 
> collection of tests, called a "suite".  The Python doctest module can 
> also create suites from embedded and external doctests.  (Embedded = 
> doctests in docstrings, external = doctests in text files.)  If you 
> write a function that returns a test suite, then that function can be 
> run with unittest.py.  For example, here's an excerpt from a unit test 
> module in PyProtocols  (protocols.tests.test_dispatch):
>
>     from unittest import TestCase, makeSuite, TestSuite
>     import doctest
>
>     # actual test classes omitted for brevity
>
>     TestClasses = (
>         TestGraph, TestTests, ExpressionTests, SimpleGenerics, 
> GenericTests,
>     )
>
>     def test_combiners():
>         return doctest.DocFileSuite(
>             'combiners.txt', optionflags=doctest.ELLIPSIS, 
> package='dispatch',
>         )
>
>     def test_suite():
>         return TestSuite(
>             [test_combiners(), ] +
>             [makeSuite(t,'test') for t in TestClasses]
>         )
>
> The two functions here, 'test_suite' and 'test_combiners' are 
> functions that return unittest "test suites".  One of them creates an 
> external doctest that runs the tests in 'combiners.txt' (a tutorial 
> text file), and the other creates a test suite that combines that test 
> with test suites for each of the other test classes in the module.
>
> Thus, to run all 50 unit tests specified by this test module, I can 
> now use:
>
>     unittest.py protocols.tests.test_dispatch.test_suite
>
> Or to run just the doctest, I can use:
>
>     unittest.py protocols.tests.test_dispatch.test_combiners
>
> And of course I can still also specify an individual test case class 
> or method.
>
> There are also a couple of other ways to gather tests.  For example, 
> this code defines a function that returns a suite of all the test 
> classes found in every module of the 'repository.tests' package:
>
>     from unittest import defaultTestLoader, TestSuite
>     import repository.tests
>
>     testNames = """
>     TestAlias
>     TestBZ2
>     TestBinary
>     TestDeepCopyRef
>     TestDelete
>     TestIndexes
>     TestItems
>     TestKinds
>     TestLiteralAttributes
>     TestMerge
>     TestMixins
>     TestMove
>     TestPerfWithRSS
>     TestPersistentCollections
>     TestRedirectToOrdering
>     TestRefDictAlias
>     TestReferenceAttributes
>     TestRepository
>     TestRepositoryBasic
>     TestSkipList
>     TestText
>     TestTypes
>     """.split()
>
>     def test_suite():
>         return TestSuite(
>             [defaultTestLoader.loadTestsFromName(name, 
> repository.tests)
>                 for name in testNames]
>         )
>
> Of course, the names in 'testNames' could name individual test 
> classes, methods, or suite-generating functions.  They can also be 
> absolute module names instead of ones relative to a particular 
> location.
>
> So as you can see, there are a lot of potential ways we could organize 
> this, with different tradeoffs depending on what tests people want to 
> run and how often, not to mention how often they need to change the 
> list of the tests.  There are even tradeoffs depending on how many 
> tests you put in a single module.
>
> Currently, I notice that most Chandler tests have only one test class 
> per module, and many of those actually only have one test method per 
> test case.  I'm guessing that this is a byproduct of how long tests 
> take to run, coupled with the fact that hardhat (unlike unittest.py) 
> doesn't allow selecting an individual test within a module to run.
>
> By contrast, using unittest.py to select and run individual tests, you 
> can actually include more tests in the same module, with no overhead 
> penalty for doing so.  This will be important as we add fast-running 
> unit tests, as each unit test tests only a very small piece of 
> functionality, and there are therefore usually a lot of them.  For 
> example, PyProtocols has only seven test modules, but these contain 
> hundreds of test methods; a typical run of those tests involves maybe 
> 320 individual test methods, each with an average of maybe half a 
> dozen assertions being tested.
>
> So, if we start adding unit tests (in the strict isolationist sense) 
> to increase the coverage-to-runtime ratio of Chandler's tests, it's 
> likely that we'll tend towards putting more tests per class and more 
> classes per module than is currently done.  So, if we set up a 
> test-gathering strategy that works well for the current situation, it 
> won't necessarily continue to work long-term.
>
> On the other hand, I don't want to burden anybody with a 
> test-gathering style that would work well in the future, but which 
> might seem tedious for today's more limited needs.  So, it seems like 
> the best thing for me to do is throw all this out here for discussion 
> and find out what approach the team would prefer to take.
>
> (P.S. I forgot to mention...  right now 'application.tests' isn't a 
> package, so it needs an '__init__.py' if you want to use 'unittest.py' 
> to run any of the tests in it.    I think all the other test locations 
> are already packages.)
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> Open Source Applications Foundation "Dev" mailing list
> http://lists.osafoundation.org/mailman/listinfo/dev
>
----
Ted Leung                 Open Source Applications Foundation (OSAF)
PGP Fingerprint: 1003 7870 251F FA71 A59A  CEE3 BEBA 2B87 F5FC 4B42