About memory ownership in SWIG and Python

Today, a long-standing bug in libyui-bindings was finally fixed.

About libyui and libyui-bindings

libyui-bindings provides SWIG based bindings for Ruby, Python and Perl to access libyui functionality. libyui is the YaST user interface library, allowing to write dialogs in a generic way so they can be displayed in text (ncurses) or graphical (Qt) environments.

Reference counting

The core of the problem is libyui relying on YaST data structures. These data structures implement reference counting, so one doesn't have to release allocated memory explicitly. That all fine, as long as you use YaST's YCP programming language to code dialogs. But coding YCP is not necessarily something a lot of developers are looking forward to.
Thats where libyui-bindings enter stage and remove the YCP restriction. I've blogged about it ealier and Jan-Simon was especially active in providing lot of Python examples.

Competing on memory ownership

Now Python is a nice scripting language but has the bad habit of doing reference counting itself. This makes coding Python extensions in C quite awkward and clashes with libyui. Speaking of bad habits, libyui also has one - claiming ownership of memory passed as pointers to some functions.
Fun starts when freeing memory. libyui and Python enter a nasty fight which Linux ends with a SIGSEGV. Jan-Simon reported this as a bug and provided a nice test case.

Swig typemaps

After some googling and reading the SWIG documentation a couple of times, SWIGs DISOWN typemap was the solution. Documentation for this is sparse and well hidden within the Chicken language chapter. Grrr.
SWIG typemaps apply to function parameters and match the parameter type and name. One cannot specify per-function typemaps, only per-parameter. Fixing the bug required libyui to explicitly name parameters claiming ownership of memory. All such parameters within the libyui API are now ending with _disown.

Solving the case

Applying the typemap is done using the %apply directive:

%apply SWIGTYPE *DISOWN { YItem *item_disown };
%apply SWIGTYPE *DISOWN { YEvent *event_disown };
%apply SWIGTYPE *DISOWN { YTableCell *cell_disown };
%apply SWIGTYPE *DISOWN { YWidgetID *newId_disown };
%apply SWIGTYPE *DISOWN { YTableHeader *header_disown };
%apply SWIGTYPE *DISOWN { YTableHeader *header_disown };
%apply SWIGTYPE *DISOWN { YWidget *parent_disown };

Now Python gives up ownership for structures passed through a _disown parameter to libyui and all libyui Python examples run without coredumps. Case closed.

Report from Fourth Annual Management Developers Conference

The fourth Management Developers Conference (mandevcon) took place November 17 - 20 2008, in Santa Clara, California.

This is the annual get together of people involved and interested in systems management technologies and standards as defined by the Distributed Management Task Force (DMTF).

The DMTF is an industry organization defining systems management standards such as the Common Information Model (CIM) or Web Services for Management (WS-Management).

For my employer, these standards play an important role as they're part of the Novell/Microsoft technical collaboration agreement about interoperable systems management.

Compared to last year, attendance to the conference was very low. I guess only about a third of last years attendance numbers were reached. This decline was mostly attributed to slashed travel budgets due to the tough economy. However, hallway conversations also hinted towards lack of scope of the conference, trying to be everything for everyone. A split into research, development and deployment topics is probably needed.

Presentations during the three days of the conference are divided into three parallel tracks. These were

• the DMTF industry initiatives (http://www.dmtf.org/initiatives)
  like
  • SMI (storage management)
  • CDM (diagnostic)
  • CMDB (configuration database)
  • Virtualization
  • SMASH (server hardware)
  • DASH (desktop hardware)
• CIM core technology
• Web Services for Management
• Developers

The full conference schedule can be obtained from here.
Sadly, the presentation slides (mostly PDF) are only available to attendees (authorization needed).

Being busy with my own slides, I wasn't able to attend as many presentations as I wanted. But I still made it to the following presentations of the first day:

Python CIM providers

Writing CIM instrumentation has a long history of being complicated and needing C or C++. Bart Whiteley of Novell presented pywbem, showing how to write CIM providers in a modern scripting language. Pywbem also comes with a provider generator, spitting out a Python skeleton provider from the definition of its CIM classes.

Multi-Tenant CIMOM, Fact or Fiction

There are a couple of CIM object managers (CIMOMs) available, mostly as open-source. However, CIM providers are mostly CIMOM-specific and can't be easily ported. With the Common Manageablity Programming Interface (CMPI), a standard ABI for CIM providers is established. Still configuration information required to install a provider is different from CIMOM to CIMOM.

This presentation showed typical obstacles in writing portable providers and gave rules and recommendations to make a provider cimom-agnostic.

cmpi-bindings - Compiler Free Provider Development

This was the first of my own presentations. Cmpi-bindings serves a similar purpose like pywbem, freeing CIM provider developers from the need for C or C++.

cmpi-bindings uses SWIG to generate bindings between the Common Manageablity Programming Interface (CMPI, the ABI between a CIMOM and a CIM provider) and a scripting language. Currently supported are Python, Ruby and Perl. Any other SWIG supported language can be added fairly easily. cmpi-bindings use SWIG in an innovative way, allowing to call into the scripting language from the CIMOM. This effectively reverses the typical flow of control for SWIG, which is calling a C or C++ library from a scripting language.

Testing with PyWBEM

Automated testing of CIM providers is often neglegted and developers use plain CIM browsers to explore their code.

Tim Potter showed various ways for testing providers with the help of Python and pywbem. Most interesting was learning about Twisted, a Python networking framework for asynchronous programming. This stuff makes your brain hurt and the O'Reilly book title illustration is worth a thousand words ;-)

---

On day 2, I had the opportunity to attend Microsofts 'Strategic Architect Forum' in San Francisco. So I skipped the mandevcon presentations for that day. I'll blog later about the Microsoft conference.

---

The last day had a complete track on WS-Management.
It started with a generic introduction by Josh Cohen from Microsoft. Josh is leading the WS-Management work group within the DMTF and the driving force behind Microsofts adaption of an open standard for systems management. Too bad his presentation isn't available online.

Openwsman swig bindings

Anas Nashif, of AutoYaST fame, showed the Openwsman swig bindings Openwsman is an open source implementation of the WS-Management protocol, used e.g. by Microsoft Windows for remote systems management.

These bindings allow for easy ws-management client programming using one's favorite programming language. Fully supported currently are Python, Ruby and Java (yuck!).

Scripting openwsman server plugins

WS-Managment is a generic protocol for resource management. Each resource is identified by a universal resource identifier (URI), which onsists of a namespace and a resource part. The openwsman server maps namespaces to plugins and lets the plugin handle the specific resource implementation.

Openwsman includes default plugins for the generic Identify operation and for CIM operations (WS-CIM). Writing plugins in C is easy, using a scripting language makes it even easier.

My presentation shows how the problem was approached and describes the implementation. It is in an alpha state currently and needs some more work to become fully functional. This will also require some changes in the openwsman plugin API to fully support plugins.
The code is part of the openwsman svn repository and available here.

ActiveWSMAN

One thing that bothers me when coding against the openwsman client API is the amount of protocol knowledge needed. Doing a simple enumeration call requires one to know about the enumerate, pull and release protocol functions. Not to mention keeping track of the context.

Having enjoyed the ActiveRecord pattern as implemented in Ruby On Rails, I collected some thoughts around using this for WS-Management client programming.

The presentation shows a concept on how to make a developers life much easier. It focuses on resources and their semantics, completely hiding the internals of the ws-management protocol.

A proof-of-concept implementation based on ActiveRecord exists and I itend to complete it in the next couple of weeks.