openage C++ <-> Python interface

openage consists of python modules, which contain the program entry point, and the library libopenage.so, which contains all C++ code.

Cython is used for glue code.

Cython crash course

Python with types

Cython modules are written in .pyx files, and roughly equivalent to .py files. In addition to regular Python syntax, .pyx files allow you to define typed functions and objects:

cdef int square(int x):
    return x * x

cdef cppclass Rectangle:
    int h, w

    int size():
        return this.h * this.w

def foo():
    cdef Rectangle r
    r.h = 5
    r.w = 6
    return square(r.size())

More language information is in the official documentation.

.pyx files are translated to .cpp by Cython (“cythonized”) as part of the openage build process; syntax errors are shown in this step.

Each .cpp file is then compiled to a Python extension module, which may be used from everywhere.

• def functions and cdef class classes can be used from Python

• cdef functions can’t be used directly

  • suitable for storage in a C function pointer

  • callable from def functions in the .pyx file

$ cython --cplus -3 test.pyx
$ g++ -shared -fPIC $(python3-config --includes) test.cpp -o test.so
$ python3
>>> import test
>>> test.foo()
900

Interface definitions

Cython can use any regular C++ function or type; for that purpose, they need to be declared in .pxd files (which are analogous to C++ .h files):

cdef extern from "<stdlib.h>":
    int atoi(const char *s)

To use this function declared in foo.pxd,

from foo cimport atoi as c_atoi

def atoi(s):
    # invokes the c implementation according to the
    # interface defined in a .pxd file
    return c_atoi(s)

pxd generation

openage has a helper, pxdgen, which auto-generates .pxd files for .h files in the libopenage/ subdirectory, from pxd: annotations in these files, as part of the build system.

The pxd annotations are really simple; just have a look at some of the headers. You’ll find the generated .pxd files next to the C++ header files.

To cimport a class Foo that was pxd-annotated in util/foo.h, type

from libopenage.util.foo cimport Foo

Cython ships lots of .pxd files for most C, C++ and CPython functions:

from libc.math cimport sin
from libcpp.vector cimport vector

cdef vector[float] vector_sin(vector[float]& args):
    return [sin(arg) for arg in args]

print(vector_sin(range(10)))

From time to time, it may be useful to have a look at the generated .cpp file, especially if you require more exotic functionality or something doesn’t work out as you expected. For that purpose, .html files are generated next to the .cpp files.

Calling C++ functions from Python

To make a C++ function available for calling from Cython code, annotate the header file with pxd comments:

libopenage/example.cpp

namespace openage {

int foo(int arg0, std::string arg1) {
        return 5;
}

}

libopenage/example.h

// pxd: from libcpp.string cimport string
#include <string>
#include "util/compiler.h"

namespace openage {

/**
 * The famous foo function. Warning: might bar occasionally.
 *
 * pxd:
 *
 * int foo(int arg0, string arg1) except +
 */
OAAPI int foo(int arg0, std::string arg1);

}

Always pxd-declare your functions as except + (“may potentially rise a C++ exception”), unless the C++ function is marked noexcept itself. If a function that is not declared except + throws anyways, the entire CPython interpreter is likely to be shredded. (Sidenote: the except + annotation enables openage-specific exception translation. To see how this is used, look in the exctranslate_tests.pyx/cpp)

To enable pxdgen, cmake must be informed about the pxd-annotated header file:

libopenage/CMakeLists.txt

pxdgen(example.h)

The function is now available from Cython. To make it available for pure-python modules, write a wrapper:

openage/foo.pyx

from libopenage.foo cimport foo as c_foo

def foo(int arg0, str arg1):
    with nogil:
        return c_foo(arg0, arg1)

openage/bar.py

from openage.foo import foo

print(foo(10, "test"))

Calling Python functions from C++

The interface works one-way: Cython can access libopenage, but not the other way round. Thus, libopenage must find its way back to Python:

• By pyinterface::PyIfFunc function pointers that are filled by Cython during initialization.

• By py::PyObj objects that hold a Python object. It may be callable.

Calling directly to C++

Any Python object can be stored in a PyObj (from libopenage.pyinterface.pyobject), which C++ can use to do calls to Python callables.

This works in the way that you create C++ objects in Cython and then call a C++ function where you pass it.

// pxd: from libcpp.string cimport string
#include <string>

// pxd: from libopenage.pyinterface.pyobject cimport PyObj
#include "libopenage/pyinterface/pyobject.h"

/**
 * pxd:
 *
 * cppclass demo_struct:
 *     PyObj obj
 *     string text
 */
struct OAAPI demo_struct {
    py::PyObj obj;
    std::string text;
};

/**
 * pxd: int cpp_function(lol_struct arg, int another_arg) except +
 */
void cpp_function(lol_struct &arg, int another_arg) {
    // native data can directly be used
    std::cout << "native_arg: " << arg.text << std::endl;

    // call the python
    // with automatic argument conversion!
    std::cout << "python call: "
        << arg.obj->getattr("py_func").call("some binary", another_arg).str()
        << std::endl;
}

from libopenage.pyinterface.pyobject cimport PyObj
from libopenage.main cimport demo_struct, cpp_function

class TestClass:
    def __init__(self):
        self.some_member = "rofl"

    def py_func(self, arg0, arg1):
        return "test: 0={} 1={} 2={}".format(self.some_member, arg0, arg1)

def entry():
    # create the object and deliver it to c++
    test_obj = TestClass()

    # python object wrapping for c++
    cdef PyObj pyobj_wrapped = PyObj(<PyObject*> some_object)

    # create a c++ object
    cdef demo_struct cppobj
    cppobj.obj = pyobj_wrapped
    cppobj.text = "behold the automatic type conversion!"

    # call to c++
    cpp_function(cpp_obj, 1337)

This means that Python calls to C++ with cpp_function, then C++ calls back to Python to the py_function.

C++ can access a PyObj in many more ways, perform casts, etc.

Registering Python functions in C++

To access call Python functions some time later, they can be remembered in C++:

Any cdef functions can be stored in C++ function pointers. The openage::pyinterface::PyIfFunc type has been created for this purpose; it allows binding arguments and makes sure that the pointer is properly initialized.

We will call this pure-Python function from C++:

openage/bar.py

def bar(arg0, arg1):
    """
    This function involves rainbows and unicorns.
    arg0 shall be an integer, and arg1 a string.
    """
    return 6.283185307179586

Declare, define and pxd-export the function pointer:

libopenage/foo.cpp

#include "foo.h"

PyIfFunc<float, int, std::string> bar;

libopenage/foo.h

// pxd: from libcpp.string cimport string
#include <string>

#include "pyinterface/functional.h"

// pxd: PyIfFunc2[float, int, string] bar
extern OAAPI PyIfFunc<float, int, std::string> bar;

Wrap the python function in a cdef function, and define a method setup(), which binds the cdef function to the PyIfFunc object.

openage/foo.pyx

from libopenage.foo cimport bar as c_bar

from .bar import bar as py_bar

cdef float bar(int arg0, string arg1) except * with gil:
    return py_bar(arg0, <str> arg1)

def setup():
    c_bar.bind0(bar)

PyIfFunc2 means that the function takes 2 arguments, and bind0 means that 0 arguments are bound (this is needed because Cython currently doesn’t support variadic template arguments).

Add a call to openage.foo.setup() to openage.cppinterface.setup.setup.

openage/pyinterface/setup.pyx

def setup():
    # (...)

    from openage.foo import setup
    setup()

    # (...)

If you forget to do that, openage.pyinterface.setup.setup will raise a fatal exception.

libopenage/bar.cpp

#include "foo.h"

std::cout << openage::bar.call(5, "test") << std::endl;

The only way of accessing Python code from C++ should be via the Func or PyIfFunc function wrappers, as those guarantee that exceptions are properly translated, among other things.

Real-life examples

• For code that wraps a C++ class for Python, see:

  • openage/cabextract/lzxd.pyx

  • libopenage/util/lzxd.h

• For code that wraps a Python class for C++, see:

  • libopenage/util/fslike/path.h

  • openage/util/fslike/cpp.pyx

• Search for .pyx files in the repo

Notes on the GIL

The GIL must be acquired for any Python functionality (even as simple as PyErr_Occurred).

However, GIL-safety is guaranteed by the combination of Cython, the pyinterface code, and the fact that libopenage doesn’t link against Python itself / include any Python headers.

Any code in libopenage can safely be run without the GIL.

Only functions that are marked with gil can be bound to PyIfFunc or Func objects; this ensures that the GIL is always re-acquired when jumping into Cython code.

Never use raw function pointers in the interface; always use the PyIfFunc or Func objects; otherwise, you’ll lose all safety guarantees.

The OAAPI macro

OAAPI marks the DLL entry-points which is necessary for the Windows build. All pxd interface functions, classes, and extern objects must be declared with OAAPI in the header file. See the samples above to understand the exact position of insertion for each of those. This can be ignored for functions defined inline in the header.