[Dev] Mock Reactor for tests?

[Phillip J. Eby]

At 02:27 PM 6/6/2005 -0700, John Anderson wrote:
>It also seems like our tests would be more valid if we were using real 
>Twisted code instead of something else.

That would be a true statement if we were only talking about integration 
tests.  However, there is a difference between unit testing and integration 
testing.  Unit testing tests components in isolation, integration testing 
tests them with other components.  Unit testing promotes development speed, 
architectural soundness, and developer confidence, while integration 
testing supports quality assurance for code release.  Unit tests should be 
very, very fast, preferably with centisecond runtimes or better for 
individual tests.  Integration tests can be slower, although being fast is 
always good.

Unit tests should always, always, always succeed, with never a broken 
checkin.  Integration tests, if they are also being used as acceptance 
tests, can be allowed to fail, because acceptance tests are used to show 
the degree of feature-completeness of a product, and in that case they 
simply reflect unfinished features.

As a general rule of thumb, unit tests should be fast enough that one does 
not feel burdened by running them after *every line or two* of new code 
written, while integration tests should be comfortably run once a day, or 
overnight.

Chandler currently has relatively few tests that qualify as unit tests; it 
needs more.  Chandler already has lots of integration tests, but they are 
no substitute for actual unit tests.

Now, even if we are talking about integration tests, note that Brian's 
tests use a "string transport", which is not "real" socket code, so by your 
argument this would not be as "valid" as using sockets.  But, if you say 
that StringTransport is "real Twisted code" because it comes with Twisted, 
then all we need to do is contribute a usable mock reactor and replacement 
fixtures to the Twisted code base, at which point it will become "real 
Twisted code".  :)

A major idea in the concept of unit tests is to test things in isolation, 
minimizing the number of external points of failure (such as external 
servers, connectivity, etc.) for the test.  Mock objects and stubs are a 
typical practice for supporting unit tests of components with external 
dependencies.  The use of interfaces to define what is expected on each 
side of such dependencies is a supporting practice.

The last time I did any significant work with Twisted, it had something 
like 6 different reactor implementations and variations thereof.  A mock 
reactor would just be #7, or #8, or how many ever such implementations they 
have now.  Twisted is built on interfaces, which means that there are 
behaviors that reactors are expected to provide and conform to, and which 
all other components of Twisted can expect, and whose limits they may not 
exceed.  This means that it is a bug in Twisted if any part of Twisted does 
not function correctly with implementation #N of the IReactor* interfaces.