D-Bus usage best practices and rapid application development enablers research project
Telepathy IPC culture and infrastructure
During years of very much trial-and-error Telepathy framework development, we've made some fundamental practical discoveries on the subject of using D-Bus and inter-process communication in general, for building complex modular object-oriented desktop and mobile applications.
These innovations include, but are not limited to
our DbusSpec extension of the D-Bus introspection XML format to a proper interface description language (IDL)
Documentation annotations which the human-readable specification is generated from, incorporated to the IDL format
the tp-glib and tp-qt4 tool chains automatically generating low-level language bindings with rich type mapping and documentation from the IDL
correctness principles on message-passing IPC usage, such as the stress on all remote method invocations being asynchronous
event-driven implementations of asynchronous calls
higher-level client-side proxy pattern to make inspecting the current state and read-only properties of remote objects convenient even given the asynchrony constraints
the feature pattern (essentially optionally enableable roles on proxy objects) to avoid causing unneeded network activity and local process wakeups
bulk object state migration from service to client patterns (informally described in e.g. AOSA book chapter section 20.2.3) to make the proxy operation more efficient
the asynchronous factory pattern, to avoid programming complexity and redundancy from a naive implementation of asynchronous stateful object proxying
Tubes which extend the application domain of D-Bus, a mostly single desktop/mobile system scope focused IPC system, over the Internet utilizing capability discovery, service announcement and firewall traversal mechanisms existing in IM networks for the purposes of media streaming (VoIP/video calls), file transfer and presence multicasting
Future directions
These discoveries have helped us make Telepathy successful numerous fields such as in the GNOME desktop (with Empathy and Gnome shell integration), in the One Laptop Per Child project and on Nokia phones (see e.g. Naba Kumar's and Mikhail Zabaluev's or Marco Barisione's presentation). There is still however lots of room for further Telepathy feature development, but implementing the aforementioned principles and patterns manually for the inter-component communication required by each new item of functionality has gotten tedious and repetitive as Telepathy has expanded.
The upstream GDBus library already implements a higher-level proxy construct complete with asynchronous method calls and rudimentary state caching. We should try to migrate the rest of our implementation support facilities, such as those based on marking some properties of objects as immutable, to GDBus and its code generator. We could also refactor some core Telepathy D-Bus communication around the ObjectManager interface which GDBus has native support for.
GDBus thus enhanced could serve as a basis for Telepathy in the future. This would enable us to have to worry less about the implementation of our IPC infrastructure, and instead focus on developing the actual user-visible features such as advances in VoIP/video calling support. But even more importantly, other projects could then benefit from our discoveries by utilizing the new GDBus functionality.
Knowledge dissemination
We feel that given the widely useful nature of our IPC usage principles, it is a shame how they have got stuck as an implementation detail of Telepathy. Some other projects (e.g. EU ACTORS, and MPRIS) have started using our IDL format and the low-level code generation tools, thankfully, but that's about it. We've also stumbled upon a lot of people hitting walls with their IPC based systems, e.g. due to impossible to solve deadlocking from trying to simulate synchronous local calling over D-Bus, or programming complexity insurmountable in growing systems from using too low-level message passing directly - which could've been avoided by understanding and adopting our principles and implementation helper mechanisms from the start.
In other words: we've learned valuable lessons during countless developer hours of tedious exploratory work on D-Bus API and wrapper library design. We don't want everybody else working on similar modular architectures to have to waste their time reinventing all this, but instead implement something new and useful which is also correct and efficient from the start.
Olli Salli is currently working on a Master's thesis which aims to put these underlying design and implementation principles of Telepathy into an academic context, linking them with bodies of research on object-oriented and distributed systems, and formalizing our extensions to, and applications of, existing concepts. However, as partially motivated above, we'd also like to
- split the thesis to a series of shorter academic and/or technical articles describing particular aspects of our innovations in a more approachable and focused way, to be formally published and thus attain wider spread of this knowledge
- further automate the code generation and enhance the library support by refactoring out common hand-written implementation patterns from tp-qt4 and tp-glib as reusable (or preferably automatically reused) skeletons
push the implementation support mechanisms as Freedesktop.org-wide de facto standards (DBus core), transcending the Telepathy specific nature - mostly through incorporating as much as possible into the D-Bus standard IDL format, and the GDBus library with its code generator
- blog, publish tutorials, host BoF sessions and workshops (Desktop Summit / GUADEC / Akademy / Linux Plumbers 2012?) on how easy building distributed software becomes with all this implementation support
Projects which could benefit from sharing infrastructure with Telepathy
Some examples of related projects using D-Bus for IPC, where we believe redundant work would be avoided and functionality and quality in general improved by migrating to an enhanced common D-Bus infrastructure:
BlueZ
- The official Linux Bluetooth stack
D-Bus interfaces promoted in a white paper
- Interfaces allow controlling the HCI daemon to configure Bluetooth adapters and security aspects like passkeys and pairing
Implements an internal DBus abstraction and a service-side object model using libdbus directly - which are redundant with even current upstream GDBus and tp-glib facilities (oddly enough, the BlueZ internal abstraction is also called gdbus!)
- The white paper examples suggest that clients should access the service objects using low-level D-Bus method calls, with no implementation support such as state-tracking/caching proxies, and in a synchronous (blocking) fashion
GNOME Keyring
- A collection of components in GNOME that store secrets, passwords, keys, certificates and make them available to applications
Based around the PKCS#11 standard
Seahorse is a frontend for managing keys and passwords
Exposes the Secret Service D-Bus interfaces for application access - also implemented by KSecretsService for cross-desktop interoperability
There is a client library abstracting the D-Bus access for applications, but it is implemented cumbersomely using raw libdbus calls and hence is hard to maintain
A replacement library supporting the full range of the functionality in the Secret Service D-Bus interfaces is in the works, using GDBus, but needs to implement a lot of state download code manually (not started yet at the time of writing) because state transfer and synchronization mechanisms, like those employed in Telepathy, are not in use
- It would be possible to backwards-compatibly retrofit Telepathy style state synchronization to the D-Bus interfaces to simplify the implementation of client libraries
NetworkManager
- Set of co-operative tools that make networking simple and straightforward
Front end <-> back end communication implemented using D-Bus with non-trivial object hierarchies
State download implemented manually without library support using direct DBus.Properties calls - this is exactly the kind of redundant work we want to avoid duplicating in projects
Closing
The grand goal of all this is to
- spread the knowledge of modern lightweight desktop IPC systems to academia (university curriculums still seem to have a focus on CORBA!) - to enable more research and up-to-date teaching in this field
- help more free software hackers to get their modular desktop software (and distributed object-oriented systems in general) built in a simple and efficient fashion with little effort, while still remaining correct ("right")

