Telepathy Connection Manager Testing Suite

Overview

We have a Twisted Python based testing suite for testing connection manager functionality. The testing suite simulates both the Telepathy client and the protocol server (e.g. Jabber or SIP server), with "tp client" or "protocol server" requesting or initiating various events and checking that the connection manager responds properly.

The suite is currently used with telepathy-gabble and telepathy-sofiasip. It consists of:

protocol-independent part that covers Telepathy part of the system (servicetest.py),
protocol-dependent part that is used for simulating protocol server (e.g. gabbletest.py in tp-gabble, sofiatest.py in tp-sofiasip)
shell scripts to make it easy to run test with the temporary DBus session bus

Operation

When executing the tests, a temporary DBus session bus is created (so we don't mess with stuff outside our test environment), connection manager is run, and then each test python script is run. For each test, the status (success or failure) is printed (based on the shell exit code), and afterwards the connection manager is shut down and the temporary session destroyed. The helper shell scripts also support running gabble under valgrind, so it is possible to do memory debugging and leak checking as well.

Each python script performes a specific test. The test is usually done in a few variants, but as the success/failure is determined from the script exit code, only the related tests (or variants of the same test) should be in one script. This helps with determining exactly which test failed.

The servicetest.py script exports a nice API for writing tests. At the moment, tp-sofiasip has older, event-based test API (with lots of expect_* functions hooking up to the interesting events), but as we found that this is cumbersone and difficult way of writing tests, we've improved it, and the new version is bundled with tp-gabble. The new API allows writing test in sequential imperative manner, which is much more understandable, and is more robust than the previous API (it's a lot easier to avoid race problems). The new API will also be ported to tp-sofiasip in the near future.

The servicetest python module is a tiny wrapper around Telepathy protocol (using directly dbus-python), exposing channel proxy creation (autodetects available interfaces), async DBus method calls and performing initial RequestConnection on the CM D-Bus object. It is assumed that the test needs only one Connection object.

The module also contains an event queue. The queue holds all events happening during the test: dbus-signal, dbus-return (from the async method calls), and, for gabble. stream-iq, stream-presence and stream-message (the XMPP stanzas that gabble sends to the server, meaning they're "incoming" in the protocol server).

The queue doesn't really store the events; it is called each time when the test script needs to wait for an event, and then simply waits for something to happen. Next, the queue matches the event that just happened to what the function specified it waits on. If there is a match, the function returns to the test script with the matched event. If the event isn't matched to anything, it's simply ignored. The function returns to the script only when there is a positive match. If there's no match within a timeout period, an exception is thrown (and the script generaly fails).

API

There are three alternative "expect" functions in the API, for different use cases:

expect() - Used when we are interested in a particular event. This is the normal use case.
expect_many() - Used when we are interested in several events, but don't care in which order they appear. This is useful when there is a possibility that some events normally appear in several possible orderings (due to the particular cpu scheduler decisions at that time), and we just want to make sure all of them happen, and catch them.
demand() - Demand that the next event that comes along be the one we're interested in (ie. don't ignore unmatched events, fail on them).

The event we're interested in is described with event type (e.g. 'dbus-signal', or 'stream-iq'), and an arbitrary number of event attributes that we know must have a certain value. Possible attributes differ for various events. Examples:

for DBus signals, attributes 'signal' (signal name) and 'args' (list of signal arguments) are defined
for DBus async method returns, attribute 'method' (method name) is defined
for gabbletest 'stream-iq' signals, 'iq_type', 'to', and possibly 'query_name' and 'query_ns' are defined

For example, to wait until Connection gets connected, we can use:

queue.expect('dbus-signal', signal='StatusChanged', args=[0, 1])

(meaning, wait until StatusChanged(status=0, reason=1) signal arrives)

Using for other connection managers

As the code is split into tp-generic and protocol-specific part, it should also be possible to use it for other connection managers, provided that there is some support in python for the underlying protocol (twisted has support for both XMPP and SIP built in, but any binding/module could be used provided it can be integrated with glib mainloop (and not e.g. block indefinitely, or stuff like that)).

Examples

There are a lot of examples of API usage in tests/ subdirectory of telepathy-gabble. Look at test-*.py files - test-connection.py is probably the simplest example. The ones having lots of expect_* functions with @match() decorator use the deprecated API. The utility shell scripts are run-with-tmp-session-bus.sh and exec-with-log.sh, also have a look at the relevant portion of Makefile.am (check-twisted target)

To run the checks, do:

make check-twisted

(optionally set environment variable GABBLE_TEST_VALGRIND to run the tests under valgrind).