Designing a RESTful api for YaST

As we move YaST to the Web, we also want to give it a web-like architecture offering its functionality for widespread consumption.

So we started off by separating the client frontend (user interface) from the server backend (functionality) and settling for a RESTful api between them. We also decided on using Ruby on Rails (usually just named 'Rails') to implement both layers. Besides its widespread adoption outside of OpenSUSE, Rails is used in a couple of company projects like

• SUSE Studio
• OpenSUSE buildservice (frontend)
• OpenFATE

Rails comes as a natural choice as we can build upon a rich knowledge base with developers sitting (literally) next door. Rails was also one of the early adopters for RESTful and offers a nice encapsulation layer named ActiveResource.

ActiveResource ...

ActiveResource gives you an object-oriented interface for resources, hiding all the HTTP protocol or serialization issues from you.

Following the model-view-controller principle of Rails, resources are separated into their model, behaviour, and visualization. The resource model is expressed as a ruby class below app/models, the resource behaviour and basic lifecycle operations (create-read-update-delete) is implemented in a Controller class living below app/controllers, and app/views hosts the visualization files.

A device resource would be modelled as app/models/device.rb, implemented as app/controllers/devices_controller.rb, and viewed through app/views/devices. (Note that controllers are always plural, even for singleton resources.)

... and routing

To make resource accessible from the outside, routing information must be placed inside config/routes.rb. Routing assigns a URL to the resource, making it accessible from the outside and uniquely identifying it.

For the device resource, a map.resources :device call inside config/routes.rb would publish the resource as /devices, effectively offering a set of URLs and HTTP methods:

HTTP verb	URL	action	used for
GET	/devices	index	display a list of all devices
POST	/devices	create	create a new device
GET	/devices/1	show	display a specific device
PUT	/devices/1	update	update a specific device
DELETE	/devices/1	destroy	delete a specific device

(taken from RailsGuides)

Limitations of the routing scheme

This scheme of identifying resources via URLs works well for a limited number of resources which all have models below app/models and controllers inside app/controllers.

But it is hard to follow if you allow resources as plugins, which you cannot control and prevent them stomping on another plugins feet. Imagine a printer plugin offering devices and a network plugin doing the same. The /devices URL cannot uniquely identify the resource, a simple

map.resources :devices

routing is not sufficient.

Pluggable resources

To make pluggable resources work, you have to centralize the routing and restrain from defining URLs. This also matches well with the RESTful principle of hypertext driven APIs and narrowing the URL exposure window.

A resource plugin for the rest-service of YaST is identified by the interface it offers and the controller implementing it.

This is done with a config file in yaml format. It specifies the interface and controller as

 interface: org.opensuse.yast.printer.devices
 controller: yast/printer/devices

in a .yml file below config/resources. A resource plugin can offer any number of resources this way.

Interfaces are qualified names like org.opensuse.yast.system.language or org.opensuse.yast.system.users. Ideally, these names should match the prefix for the PolicyKit action controlling access to the resource.

Implementation details

To minimize name clashes, the controller implementing the resource must be inside a namespace. Rails supports this by allowing subdirectories below app/controllers and automatically wrapping controller classes within modules matching the directory names.

Thus a network/device_controller and a printer/device_controller can live side-by-side an be accessed as Network::DeviceController and Printer::DeviceController. You could even nest more deeply like Org::Opensuse::Yast::System::Users, but let's not overdo it.

Rails resource routing puts another restriction on URLs: Rails manages routes as a set of maps, describing URLs as a combination of controller (implementing the resource) and action. Thus the file name and path of the controller define the URL, you cannot name them independently. So the controller: value in the resource description directly affects the resulting URL. Different resource plugins should use different namespaces to effectively prevent name clashes.

Querying the rest-service

The rest-service only publishes a single url, namely /resources, where clients can query for the URL offering a specific interface.

This URL follows RESTful principles since a HTTP GET request returns a list of all resources. Its also possible to request a specific output format, either by setting the Accept header of the http request or by appending .xml or .html.

You can easily try it out by starting the rest-service and pointing a browser to http://localhost:3000/resources.

To find the right resource URL, the client just needs to pass the interface name as ?interface=name and extract the reference from the result. Stay tuned for a future post exploring the client side in more detail.

A closer look at REST

YaST, the openSUSE installation and configuration tool, is about to get a web based user interface.

The interesting part of this is the proposed service-oriented architecture. REST, Representational State Transfer, is currently the holy grail for managing resources (objects) using http.
REST is not a protocol but an architectural style making best use of the properties of http. And the Internet is the best proof that this style performs and scales well with a distributed client/server architecture.

Roy Fielding, one of the authors of the http protocol, has written a complete doctoral dissertation on this topic. For the purpose of this blog entry, seeing it as a lightweight alternative to e.g. XML-RPC or SOAP is sufficient.

How does it relate to YaST ?

As Stefan has explained on his blog before, a web based YaST will be splitted into client and server parts. This forms a three tiered architecture, where the server runs on the managed system and exposes a REST-style API to access YaST functionality.

Getting this API right in terms of extensibility, flexibility or conformance to REST best practices is an important design goal.

Now whats good REST style ?

There is no fixed list of requirements for REST or a conformance testing tool. Designing a well behaved REST implementation is mostly about learning from others experience and follow common practice.

Browsing through popular REST related bookmarks at delicious.com gives me a much better signal-to-noise ratio than Google. And it pointed me to a couple of useful references for the do's and dont's when planning a REST-style architecture.

1. REST APIs must be hypertext-driven
2. Versioning REST Web Services
3. Common REST Mistakes

Hypertext style

The first is the most interesting (IMHO). Its about exposing only a very limited set of explicit URIs. This also prevents hard-coding them into the client but let the client query the server instead. So you start from a single URI and hop from there (thats what hypertext is all about!) to the right resource. Thats like moving from node to node in a state diagram.

Versioning in media types

When doing API versioning on the web, one might be tempted to do an all-or-nothing approach and embed a version specifier (like .../v1/...) into the URL. The second link above explains why this is a bad idea and comes up with a better one: media types. The client can tell the server how to format the response to an http request by setting the Accept field in the http header. Its like calling a function and telling it the expected return type.

And REST explicitly supports this style as it does not make any assumption on the representation of a resource. Client and server are free to agree on the actual format. Using XML for object serialization is a smart move as it allows for extensibility. Clients just pick the xml tags they need and ignore all others.

Stateless

REST, being based on http, stipulates a stateless protocol. Doing stateful REST is listed as one of the typical mistakes in implementing REST-style.
Considering the proposed YaST web architecture, this means keeping state (session) information in the client and not in the server.

More recommended reading

• How to Create a REST Protocol
• Explaining REST to Damien Katz
• Roger L. Costello: REST

Satsolver bindings documentation available

One of the outstanding features of openSUSE is the Sat solver for package dependency resolution.

Somewhat unknown to most developers, Satsolver provides bindings for the three major scripting languages: Perl, Python and Ruby. The bindings provide an easy way to learn about the Satsolver API, to explore package metadata and play what-if scenarios installing, updating and deleting packages.

The bindings are created using SWIG together with an application layer on top of the raw libsatsolver. The focus is on usability and support of the scripting language, while hiding internals of the Satsolver implementation.

The bindings documentation are hosted at http://www.suse.de/~kkaempf/satsolver-bindings-ruby for now. I hope to find a permanent place at opensuse.org in the future.

The documentation is created using rdoc with a SWIG specific parser. rdoc comes with Ruby, just install the ruby package for the rdoc framework. The swig parser is currently hosted inside the Satsolver git repo.

Although based on a Ruby specific tool, the use of SWIG ensures portability across languages. Example uses of the bindings are available for Perl, Python and Ruby If you need Satsolver bindings for other languages, drop the ZYpp developers a mail.

D-Bus service on demand

Here is the last part of my mini-series about D-Bus, this time on D-Bus activation

In my last blog entry I talked about implementing a D-Bus service in Ruby. Running the service was done as a background daemon, started by root (or with other credentials).

D-Bus also provides an on-demand service creation, named activation. Here the D-Bus daemon creates the service as soon as a client is requesting it. Shutdown of the service is at the discretion of the implementation, the D-Bus deamon does not automatically deactivate a running service.

About D-Bus activation

For D-Bus actication, you have to create a .service file. This is an INI-style format file containing

• the name of the service
• the file to run, implementing the service
• the user name to run the service as

Since the D-Bus daemon runs as user 'messagebus' and thus is unprivileged. It cannot directly run the service or assume the correct user credentials.

Instead, it uses a service helper which is setuid root to start the service. This helper is configured within /etc/dbus-1/system.conf as <servicehelper> and typically resides below /lib/dbus-1/dbus-daemon-launch-helper (/lib64/dbus-1/dbus-daemon-launch-helper on 64bit systems).

dbus-daemon-launch-helper is extra-picky when it comes to security and checks its own configuration at startup. If your D-Bus service fails to get activated, maybe dbus-daemon-launch-helper is not correctly installed.

Writing the .service file

The name of the service file must be equal to the service name with a .service extension.

A typical service file for my.awesome.Service would be named my.awesome.Service.service and contain

# DBus service activation config
[D-BUS Service]
Name=my.awesome.Service
Exec=/usr/bin/awesome
User=the_dude

Name is the name of the service, Exec the path to the executable implementing the service and User the user name to run the service. Be careful when using User=root, only very few services actually need root rights.

The Exec line can also contain arguments to be passed to the executable. This can be used to e.g. implement multiple services in a single binary and pass the actual service name as a parameter.

Installing the .service file

Just copy the service file to either /usr/share/dbus-1/system-services (for a system bus service) or to /usr/share/dbus-1/services (for a session bus service).

Your service is now ready to run. The D-Bus daemon will automatically pick up changes in the service directories and find the new file.

Driving the D-Bus with Ruby

Having looked at the D-Bus from the client perspective before, its now time to get behind the wheel.

In order to drive the D-Bus, you need to decide on a couple of things:

• which bus to drive
  you can choose between the system(-wide) bus or the (per-)session bus
• the service name
  this is the name other clients can find your service, the name has dotted notation, e.g. my.awesome.Service
• the objects to publish
  each service can provide any number of objects, denoted a slash-separated path name, e.g. /my/awesome/Service/thing
• the interface name
  methods offered by objects are grouped by interface name. Usually its used to flag object capabilities. Interface names have dotted notation, usually prefixed by the service name, e.g. my.awesome.Service.Greeting
• the methods
  these are the exported functions of the objects. Methods, also called members in D-Bus speak, have input parameters and a return value.

Obtaining a drivers license

Allowance for driving the D-Bus is controlled by config files below /etc/dbus-1. /etc/dbus-1/system.conf controls the system bus, /etc/dbus-1/session.conf controls the session bus. Both load additional configuration files from sub-directories (/etc/dbus-1/system.d/ resp. /etc/dbus-1/session.d/)

These config files are xml-based busconfig snippets, defining policies on who's allowed to own a service who can use which interfaces of the service.

A typical service configuration looks like this

 <!DOCTYPE busconfig PUBLIC "-//freedesktop//DTD D-BUS Bus Configuration 1.0//EN"
   "http://www.freedesktop.org/standards/dbus/1.0/busconfig.dtd">
 <busconfig>
   <policy user="root">
     <allow own="my.awesome.Service" />
     <allow send_destination="my.awesome.Service" />
   </policy>
   <policy context="default">
     <allow send_destination="my.awesome.Service"
            send_interface="my.awesome.Service.Greeting" />
     <!-- introspection is allowed -->
     <allow send_destination="my.awesome.Service"
            send_interface="org.freedesktop.DBus.Introspectable" />
   </policy>
 </busconfig>

It gives root the right to own the service my.awesome.Service and to use it as a client. Any other user can only use the my.awesome.Service.Greeting interface of my.awesome.Service and can introspect it.

Convention tells you to name this busconfig file after your service with a .conf extension, e.g my.awesome.Service.conf. Place it below /etc/dbus-1/session.d/ or /etc/dbus-1/system.d/, depending which bus you want to drive.

Coding a D-Bus service

Examples for creating a D-Bus service using C are scarce and scary. Esp. programming in C while avoiding Glib, things become really ugly.

As we will see, using the ruby-dbus extension to Ruby makes creating D-Bus services a breeze.

We start with creating a DBus::Object defining (dbus_)methods under a (dbus_)interface.

#!/usr/bin/env ruby

require 'dbus'

class Awesome < DBus::Object
  # Create an interface.
  dbus_interface "my.awesome.Service.Greeting" do
    dbus_method :hola, "in name:s" do |name|
      puts "Hola #{name}!" 
    end
    dbus_method :hello, "in name:s, out res:s" do |name|
      "hello #{name}!" 
    end
    dbus_method :ahoj, "in name:s, in title:s" do |name,title|
      puts "Ahoj #{title} #{name}!" 
    end
  end
end

The dbus_method call get the method name passed as a symbol, followed by its signature. The signature describes the input ("in") and output ("out") parameters by name (before the colon) and type (after the colon). The parameter names are just syntactic sugar, but the type is essential and enforced by D-Bus. Look at D-Bus type signatures to find out how to specify types.

ruby-dbus uses the method name and signature to automatically generate an XML representation of the interface needed for D-Bus introspection. This allows clients to find out about available interfaces and their methods at runtime.

The following do ... end block contains the method implementation. Every "in" parameter is passed as a value to the block in typical ruby fashion.

Next is connecting to the bus and obtaining the service. You can easily export multiple services if the bus configuration allows that.

# Choose the bus (could also be DBus::session_bus)
bus = DBus::system_bus
# Define the service name
service = bus.request_service("my.awesome.Service")

Then we create the object and export it through the service.

# Set the object path
obj = Awesome.new("/my/awesome/Service/thing")
# Export it!
service.export(obj)

Note that there is no requirement to create and export objects in advance. You can start with a simple object and method, creating additional objects as requested by the client. Hal is a typical example for such a service.

Finally we start the listener and wait for incoming requests.

# Now listen to incoming requests
main = DBus::Main.new
main << bus
main.run

main.run never returns, as you would expect from a service daemon. Either kill(1) it, code a timeout signal or implement a method calling exit.

Running the service

Running the D-Bus service is as easy as running any other Ruby program:

kkaempf> ruby my_service.rb

This, however, will get you a Exception `DBus::Error' at (eval):26 - org.freedesktop.DBus.Error.AccessDenied: Connection ":1.967" is not allowed to own the service "my.awesome.Service" due to security policies in the configuration file

Looking at our busconfig defined above immediately reveals the error. Only user root is allowed to own the service. So we better do a

kkaempf> sudo ruby my_service.rb

to get this running.

Now congrats for passing your drivers exam ! ;-)

Riding the D-Bus with Ruby

The last time I looked at D-Bus is a couple of years ago. What I saw back then was promising in technology but ugly in (C-)programming. D-Bus has come a long way 'til then.

And so have my programming skills with scripting languages, esp. with Ruby. The ruby-dbus project provides a nice and easy-to-use programming API, once one has mastered the lack of examples and the D-Bus nomenclature.

About D-Bus

There is plenty of information available on D-Bus. I personally found the Introduction to D-Bus most valuable from a developers point of view.

D-Bus originates from the freedesktop.org initiative and is hosted at www.freedesktop.org

Basically, D-Bus is a RPC (remote procedure call) mechanism, allowing different programs to talk to each other and provide services in a standardized way.

The transport used for talking is called a bus, meaning everyone can ride (connect to) it. Usually there are two independent buses available

• System bus
  This is for system-wide services, like hardware information (usually provided by HAL)
• Session bus
  This is for per-login services, like a Gnome Desktop session.

Talking is done in a client/server fashion. The server code connects to the bus offering a service to be used by the clients. In order to find each other, services are names in a dotted syntax. E.g. org.freedesktop.Hal is offered by the HAL daemon providing hardware information.

The nice thing about D-Bus is that it allows introspection. You can ask a service about its capabilities. And the D-Bus itself is a service, so you just need to know about org.freedesktop.DBus to find all other services.

Services provide objects. These are organized in a tree-like fashion and typically addressed using slash-separated paths, just like filenames. Iterating over the objects of org.freedesktop.Hal gives you all devices.

Objects then have members. Members are methods to call, providing the functionality of the service. To make things more complicatedstructured, members are grouped in interfaces. Interfaces are comparable to capabilities of an object. For example the Hal object /org/freedesktop/Hal/devices/storage_model_DVDRW_LH_20A1S has the org.freedesktop.Hal.Device.Storage.Removable interface with the bool CheckForMedia() method.

Putting it all together gives you this chain
Bus (System/Session)
   -> Service (e.g. org.freedesktop.Hal)
   -> Object (e.g. /org/freedesktop/Hal/devices/storage_model_DVDRW_LH_20A1S)
   -> Interface (e.g. org.freedesktop.Hal.Device.Storage.Removable)
   -> Member (e.g. bool CheckForMedia())

There's more to D-Bus, like signals and signatures and activation and ... follow the documentation links given above if you want to learn it all. The explanation up to now should be sufficient to understand the basics of the ruby-dbus interface, however.

Using ruby-dbus

Connecting to the bus in Ruby is as easy as

require 'dbus'

bus = DBus::SystemBus.instance
# resp. 'bus = DBus::SessionBus.instance'

DBus::SystemBus is a Singleton, hence the .instance instead of the usual .new for creating the object.

Now you can create a proxy object. Its named 'proxy' because the real object lives on the other side of the connection, in the service. The proxy object is in your application and proxy-ing calls via D-Bus to the service object. You can use this to find all services on the bus

require 'dbus'

bus = DBus::SystemBus.instance

bus.proxy.ListNames[0].each do |service|
  puts "Service: #{service}"
end

Given a known service, D-Bus introspection allows to find its objects, subnodes and interfaces

require 'dbus'

bus = DBus::SystemBus.instance

# Create the proxy object
proxy = bus.introspect "org.freedesktop.Hal", "/"

# proxy.bus gives you the bus
# proxy.path is the object path
# proxy.destination is the service name

# Print object interfaces
proxy.interfaces.each do |interface|
  puts "Object #{proxy.path} provides #{interface}"
end

# Print object subnodes
proxy.subnodes.each do |path|
  puts "-> #{proxy.path}/#{path}"
end

A specific interface of an object can be accessed by the [] operator. And the interface knows about the signature of its methods.

require 'dbus'

bus = DBus::SystemBus.instance
# create proxy for the 'computer' device
proxy = bus.introspect "org.freedesktop.Hal", "/org/freedesktop/Hal/devices/computer"

# Print object interfaces
proxy.interfaces.each do |interface|
  puts "Object #{proxy.path} provides #{interface}"
  proxy[interface].methods.each do |key,value|
    puts "  #{value.rets} #{key}( #{value.params} )"
  end
end

Due to the dynamic nature of Ruby, Object methods are directly accessible in normal Ruby conventions. One just has to select the right interface first.

require 'dbus'

bus = DBus::SystemBus.instance
proxy = bus.introspect "org.freedesktop.Hal", "/org/freedesktop/Hal/devices/computer"

iface = proxy["org.freedesktop.Hal.Device.CPUFreq"]
freq = iface.GetCPUFreqPerformance
gov = iface.GetCPUFreqGovernor
puts "Frequency #{freq}, Governor #{gov}"

Riding the D-Bus with Ruby is easy and fun !

About memory ownership in SWIG and Python

Today, a long-standing bug in libyui-bindings was finally fixed.

About libyui and libyui-bindings

libyui-bindings provides SWIG based bindings for Ruby, Python and Perl to access libyui functionality. libyui is the YaST user interface library, allowing to write dialogs in a generic way so they can be displayed in text (ncurses) or graphical (Qt) environments.

Reference counting

The core of the problem is libyui relying on YaST data structures. These data structures implement reference counting, so one doesn't have to release allocated memory explicitly. That all fine, as long as you use YaST's YCP programming language to code dialogs. But coding YCP is not necessarily something a lot of developers are looking forward to.
Thats where libyui-bindings enter stage and remove the YCP restriction. I've blogged about it ealier and Jan-Simon was especially active in providing lot of Python examples.

Competing on memory ownership

Now Python is a nice scripting language but has the bad habit of doing reference counting itself. This makes coding Python extensions in C quite awkward and clashes with libyui. Speaking of bad habits, libyui also has one - claiming ownership of memory passed as pointers to some functions.
Fun starts when freeing memory. libyui and Python enter a nasty fight which Linux ends with a SIGSEGV. Jan-Simon reported this as a bug and provided a nice test case.

Swig typemaps

After some googling and reading the SWIG documentation a couple of times, SWIGs DISOWN typemap was the solution. Documentation for this is sparse and well hidden within the Chicken language chapter. Grrr.
SWIG typemaps apply to function parameters and match the parameter type and name. One cannot specify per-function typemaps, only per-parameter. Fixing the bug required libyui to explicitly name parameters claiming ownership of memory. All such parameters within the libyui API are now ending with _disown.

Solving the case

Applying the typemap is done using the %apply directive:

%apply SWIGTYPE *DISOWN { YItem *item_disown };
%apply SWIGTYPE *DISOWN { YEvent *event_disown };
%apply SWIGTYPE *DISOWN { YTableCell *cell_disown };
%apply SWIGTYPE *DISOWN { YWidgetID *newId_disown };
%apply SWIGTYPE *DISOWN { YTableHeader *header_disown };
%apply SWIGTYPE *DISOWN { YTableHeader *header_disown };
%apply SWIGTYPE *DISOWN { YWidget *parent_disown };

Now Python gives up ownership for structures passed through a _disown parameter to libyui and all libyui Python examples run without coredumps. Case closed.