[Cosmo] Writing tests in silmut

Heikki Toivonen heikki at osafoundation.org
Thu Feb 2 15:45:40 PST 2006

The ticket and caldav compliance testsuite silmut is written in Python
and uses only the Python stdlib to do its thing. (I think it should have
some XML support, which will come as ElementTree module which will be
included in Python stdlib in Python 2.5.)

The tests themselves are written using the Python doctest module
(http://docs.python.org/lib/module-doctest.html). Doctest allows you to
mix documentation and code samples in a way that you can run the code
samples automatically, and make sure that the code really did run the
way you expected. It's a great tool for unit tests, because you can
write the unit test in the function docstring both as an example with
the added benefit of running the tests as part of normal unit tests.

Currently silmut is trivially simple. There are some helper functions
that let you type less. You write in the command you want to send to the
server, and what you expected the server to send back. When you run the
file it will be quiet if everything worked, but will show you exactly
what differed from expected if things worked.

So let's see how you could add a test for MKTICKET:

   Here's some initialization for stuff that I am using in other tests
   as well, but that are also needed in this one. user1, password1, path
   are variables provided by the framework:

    >>> auth = 'Basic %s' % base64.encodestring('%s:%s' % (user1,
    >>> authHeaders = {'Authorization': auth}
    >>> minTicket = """<?xml version="1.0" encoding="UTF-8"?>
    ... <X:ticketinfo xmlns:D="DAV:"
    ... <D:privilege><D:read/></D:privilege>
    ... <X:timeout>Second-60</X:timeout/>
    ... </X:ticketinfo>"""
    >>> home1 = '%s/home/%s' % (path, user1)

    Now this is the actual test. request() is a helper function provided
    by the framework. You give it a method, a path, a body and a
    dictionary of headers:

    >>> r = request('MKTICKET', home1, body=minTicket, headers=authHeaders)

    r now holds the response object. One member of that is the HTTP
    status code, which is what I want to test here. So let's just
    print the value and in the next line write the value we expected;
    the # is a comment and will just help me figure out which test
    failed (even though I will also get a line number):

    >>> r.status # MKTICKET OK

Notice my prose is mixed with the code. The code lines all start with
>>> or in the case of a continuation line with ...

Now we run ./src/test/bin/silmut.py and hope that it doesn't say
anything (all is good):

File "./src/test/bin/silmut.py", line 148, in __main__.ticket
Failed example:
    r.status # MKTICKET OK

Oops, something wrong! 400 is bad request. I check what I wrote above
and notice I made a typo with the XML (notice erroneous / in
</X:timeout/>). I fix that, rerun, and get no output, meaning all tests

Sometimes when I am writing the test I am not exactly sure what the
return values should be. So I could have omitted the 200 above and I
would have received an error about getting response 400 while nothing
was expected. I would still need to figure out if this made sense, of

Another benefit of doctests is that you can copy and paste code from
python console into doctest. For example, I could launch python from
shell and, I would get the prompt:


Now, let's import silmut (need to be in the same dir as silmut):

>>> from silmut import *

Now I have all the framework available to me (with default settings):

>>> host
>>> host = ''

Let's do something:

>>> r = request('GET', '/')
>>> r.status

Now I could just copy and paste the last three lines from the shell to
silmut and it would automatically execute as part of the suite.

That's still more writing than I'd like to do for each test, so I am
working on reducing even that. Also it would be nice if it automatically
parsed all the XML both sent and received.

  Heikki Toivonen

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 249 bytes
Desc: OpenPGP digital signature
Url : http://lists.osafoundation.org/pipermail/cosmo-dev/attachments/20060202/3650e8f6/signature.pgp

More information about the Cosmo mailing list