[Dev] Unit test runner

[Phillip J. Eby]

Hi folks.  Lisa has asked me to put together a unit test runner that will 
allow selective running of multiple unit tests, for example to run just 
fast unit tests, or to run all tests including functional tests.  I have a 
couple of different ways to do that, but I'd like to get some input from 
the team as to what they'd like to see in such a runner.  But before I get 
to the questions, I need to lay out some background on Chandler's current 
test setup, and the options that are available for how to proceed.

Currently, the way most of Chandler's tests are structured, you can run 
individual test modules with either hardhat or unittest.py, e.g.:

     unittest.py repository.tests.TestText

will run that specific test module, or:

     unittest.py repository.tests.TestText.TestText.testBZ2Compressed

to run an individual test within the module.  You can also use hardhat to 
collect all test modules named 'Test*.py'.

So, I gather that what Lisa's asking for is the ability to also run a 
selected subset of the total collection of tests, such as just the "strict" 
unit tests that test subsystems in isolation, with individual runtimes of 
about 10ms or less.  (Most of Chandler's current tests are technically 
speaking functional or integration tests, as they do not test subsystems in 
isolation from each other.  These types of tests are important also, but 
because they take longer to run, it's useful to also have more 
"isolationist" tests for use during development.)

Python's unittest module has a standard way of representing a collection of 
tests, called a "suite".  The Python doctest module can also create suites 
from embedded and external doctests.  (Embedded = doctests in docstrings, 
external = doctests in text files.)  If you write a function that returns a 
test suite, then that function can be run with unittest.py.  For example, 
here's an excerpt from a unit test module in 
PyProtocols  (protocols.tests.test_dispatch):

     from unittest import TestCase, makeSuite, TestSuite
     import doctest

     # actual test classes omitted for brevity

     TestClasses = (
         TestGraph, TestTests, ExpressionTests, SimpleGenerics, GenericTests,
     )

     def test_combiners():
         return doctest.DocFileSuite(
             'combiners.txt', optionflags=doctest.ELLIPSIS, package='dispatch',
         )

     def test_suite():
         return TestSuite(
             [test_combiners(), ] +
             [makeSuite(t,'test') for t in TestClasses]
         )

The two functions here, 'test_suite' and 'test_combiners' are functions 
that return unittest "test suites".  One of them creates an external 
doctest that runs the tests in 'combiners.txt' (a tutorial text file), and 
the other creates a test suite that combines that test with test suites for 
each of the other test classes in the module.

Thus, to run all 50 unit tests specified by this test module, I can now use:

     unittest.py protocols.tests.test_dispatch.test_suite

Or to run just the doctest, I can use:

     unittest.py protocols.tests.test_dispatch.test_combiners

And of course I can still also specify an individual test case class or method.

There are also a couple of other ways to gather tests.  For example, this 
code defines a function that returns a suite of all the test classes found 
in every module of the 'repository.tests' package:

     from unittest import defaultTestLoader, TestSuite
     import repository.tests

     testNames = """
     TestAlias
     TestBZ2
     TestBinary
     TestDeepCopyRef
     TestDelete
     TestIndexes
     TestItems
     TestKinds
     TestLiteralAttributes
     TestMerge
     TestMixins
     TestMove
     TestPerfWithRSS
     TestPersistentCollections
     TestRedirectToOrdering
     TestRefDictAlias
     TestReferenceAttributes
     TestRepository
     TestRepositoryBasic
     TestSkipList
     TestText
     TestTypes
     """.split()

     def test_suite():
         return TestSuite(
             [defaultTestLoader.loadTestsFromName(name, repository.tests)
                 for name in testNames]
         )

Of course, the names in 'testNames' could name individual test classes, 
methods, or suite-generating functions.  They can also be absolute module 
names instead of ones relative to a particular location.

So as you can see, there are a lot of potential ways we could organize 
this, with different tradeoffs depending on what tests people want to run 
and how often, not to mention how often they need to change the list of the 
tests.  There are even tradeoffs depending on how many tests you put in a 
single module.

Currently, I notice that most Chandler tests have only one test class per 
module, and many of those actually only have one test method per test 
case.  I'm guessing that this is a byproduct of how long tests take to run, 
coupled with the fact that hardhat (unlike unittest.py) doesn't allow 
selecting an individual test within a module to run.

By contrast, using unittest.py to select and run individual tests, you can 
actually include more tests in the same module, with no overhead penalty 
for doing so.  This will be important as we add fast-running unit tests, as 
each unit test tests only a very small piece of functionality, and there 
are therefore usually a lot of them.  For example, PyProtocols has only 
seven test modules, but these contain hundreds of test methods; a typical 
run of those tests involves maybe 320 individual test methods, each with an 
average of maybe half a dozen assertions being tested.

So, if we start adding unit tests (in the strict isolationist sense) to 
increase the coverage-to-runtime ratio of Chandler's tests, it's likely 
that we'll tend towards putting more tests per class and more classes per 
module than is currently done.  So, if we set up a test-gathering strategy 
that works well for the current situation, it won't necessarily continue to 
work long-term.

On the other hand, I don't want to burden anybody with a test-gathering 
style that would work well in the future, but which might seem tedious for 
today's more limited needs.  So, it seems like the best thing for me to do 
is throw all this out here for discussion and find out what approach the 
team would prefer to take.

(P.S. I forgot to mention...  right now 'application.tests' isn't a 
package, so it needs an '__init__.py' if you want to use 'unittest.py' to 
run any of the tests in it.    I think all the other test locations are 
already packages.)