XCB & XKB: it compiles!

While trying to autogenerate C interfaces for XKB during the last 2 months it sometimes felt like an attempt to "behead the Hydra": Whenever I fixed a problem, a number of new issues arose (deep from the intestines of the python code generator). At the moment it seems I have reached a state where no new heads emerge, and I even managed to cut off a number of the most ugliest...

The cause of most (all?) troubles is a very basic way of distinguishing data types into fixed size (everything you can feed to sizeof), variable size data types (lists, valueparams, ...) and padding. In the X protocol fixed size fields are normally grouped together, so e.g. in a request all fixed size fields are collected before any variable size fields.

XCB, whose purpose it is to translate between X server and client, takes advantage of this ordering: Several grouped fixed size fields are conveniently mapped to a C struct, so it is fairly easy to deal with. The treatment of variable size data is more difficult and involves the use of (autogenerated) accessors and iterators. Also the specific number of elements in a variable size data type can depend on expressions that are specified in the protocol, but need to be evaluated at runtime.
Now XKB breaks with the "pure doctrine" of the X core protocol: Fixed and variable size fields are intermixed in several cases and new expressions are introduced. Further problematic cases include a variable size union, lists with a variable number of variable size elements, ... And finally, the keyboard extension defines a new datatype (ListOfItems), which is referred to as 'switch' in XCB.

Switch is a kind of list, but the 'items' are not necessarily of the same type, and whether an item is included or not depends on expressions that have to be evaluated for each item separately. Defining a C mapping for 'switch' was one of the main goals of my work, and it turned out that a set of new functions was needed. These new functions must either 'serialize' a switch into a buffer, depending on the concrete switch conditions, or 'unserialize' it to some C struct. As 'switch' can contain any other data type valid in the X protocol (which means especially that it also can contain other switches...), 'serialize'/'unserialize' had to be defined for all those data types.
Once I had the autogenerator spill out '_serialize()' and '_unserialize()', it turned out they can be used to deal with some other special cases as well - most notably the (above-mentioned) intermixing of fixed and variable size fields.

But probably the most appealing feature of the serializers is that they are invisible to anyone who wants to use XCB as they get called automatically in the autogenerated helper functions.

A last note on the current status:
+ xkb.c (autogenerated from xkb.xml) compiles!
+ the request side should be finished (more tests needed however)
+ simple XKB test programs run
- on the reply side, special cases still need special treatment
- _unserialize() should be renamed to _sizeof() in some cases
- some special cases also need special accessors

The code is currently hosted on annarchy, but I will export the repos shortly.

C mapping for <switch>

Autogenerating C code from XCB's xkb.xml requires mapping of the newly introduced <switch> tag. Discussion on the mailing list (special thanks to Peter Harris & Jamey Sharp) revealed additional requirements:

• <valueparam> (a special kind of X protocol list type) can in principle be rewritten to <switch> - if one would do so in practice, the relevant request interface should not change. At the same time, auxiliary functions that reside in util/aux/ at the moment could be autogenerated from the protocol description. 
• <switch> is a special list type that allows conditional inclusion of fields. In order to achieve the mentioned compatibility with request interfaces for <valueparam>, a new set of functions is needed, _serialize() and _unserialize(), that transform an easy-to-use auxiliary data structure into a character stream and vice versa, evaluating the switch conditions in each case. 
• Requests that contain <switch> statements are best called from C using a newly introduced set of _aux() functions, that take an auxiliary data type for switch as an argument and perform the necessary call to _serialize() automatically. 

The code generator is now working for a subset of <switch> statements (ie. those that consist of fixed size fields only), the remaining cases are more difficult and thus need debugging. As a prototype, I have changed the <valueparam> in the CreateWindow request (xproto.xml) to <switch>, the overhead introduced by the additional serialization process is neglible on my machine. 

XCB-GSoC: xkb.xml & C language mapping

It's been some time since my last post, and the first C language mapping for the newly introduced xml tags has emerged in the meanwhile. But before I'll tell you about the new functionality, let me first describe a few changes I made to the existing code to fix a few issues with XCB's internals.

There are a few issues hidden deep inside the XCB C code generator that had to be changed in order to enable parsing of the XKB protocol description. Here's a short overview:

• for Types with a variable length, a length field has to be registered with the Type system. Within the context of the newly introduced tag, the lookup for the length field had to be extended to include all anchestor Types
• in some cases, no C mapping has been defined up to now for valid XML protocol descriptions, these include 1. requests with fields that are not fixed size and not lists 2. request with list of variable-sized elements (buggy) 3. structs with fixed-size fields following variable-sized fields. These issues will be addressed by changing the way Types are serialized in the C language mapping. 
• There is an interface to make value-params somewhat easier to use in the util/aux module. This interface can be made obsolete by rewriting value-params to switch (thanks to Peter Harris for pointing this out, it provides an excellent test case for <switch>). 

The changes I made to the existing code still remain to be properly documented. 

It remains to give a short overview of the mapping for the new xml tags:

• <enum> is simply mapped to the corresponding named integer constant in C
• <sumof> is mapped to a function call to xcb_sumof (newly implemented) which performs a summation of the elements of the referenced list
• <popcount> is mapped to a function to xcb_popcount
• <switch> is mapped to a series of if statements, each of the form if (switch_expr & bitcase_expr); for each a struct is generated that encompasses all named bitcase fields
• <bitcase> fields are serialized no different from other field, except they get a prefix (switch_struct->)

The new elements still need more testing, however I will start writing XKB utility functions within the next days. Any discussion of new interfaces takes place on the XCB mailing list, all comments are appreciated so please feel free to join :)

Internals of XCB python generator code

Hi, first: thanks for reading!
The Goal of my GSoC is to extend input support for XCB - i.e. implement XKB bindings and utility functions. The first thing I have to concentrate on is to generate a C language binding for the XKB protocol - the required XML protocol description has been done by Mariusz Ceier in last year's GSoC. 
The XML files describing the X protocol and extensions are converted into C code with a code generator written in python. The code generator basically consists of two parts: One part (described here in detail) parses the XML files, the other part (will be described in one of my next posts) produces the language mapping. 
I have extended the XML parser & the underlying Type system as a number of new elements are required in order to describe the XKB protocol extension. The newly introduced types still need to be documented, I will do that as soon as I am sure they are complete :)
The next step will be to think of some C mapping for the new types - this as well be the topic of one of the next posts.