Interfacing Python/numpy from C++ (Part 1/3)

Sometimes there is a need to interface Python and numpy from C++. Maybe your Python code is too slow for a specific application or you have an old C++ library that you want to run on your data. 

In this three part series we will look at how to go from Python to C++ and back again using a few different techniques. Lets start from the beginning using the Python C/C++ API. This is quite verbose and a bit tedious to write but has the advantage that there is very few prerequisites. 

For this purpose lets assume that we want to call the following function from Python: 

void cpp_add(const double *a, const double *b, double *result, int size){
for (int i=0; i!=size; ++i){
result[i] = a[i] + b[i];
}
}

Now I don’t suggest you to write code in this style, better use some nice modern C++ but for the purpose of this post it will do. So lets start out Python module with the following lines. 

#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <Python.h>
#include <numpy/arrayobject.h>
#include "function.h" //this is the function that we want to call

This says that we don’t want to use the deprecated part of the numpy API, so anything removed before v1.7 . Then we include Python.h for the Python API and arrayobject.h for numy arrays. Finally the function that we want to call is declared in function.h

//Docstring
static char module_docstring[] = "Example module providing a function to add two numbers";

//function declaration
static PyObject *add(PyObject *self, PyObject *args);

//Module specification
static PyMethodDef module_methods[] = {
{"add", (PyCFunction)add, METH_VARARGS, add_doc},
{NULL, NULL, 0, NULL}};

static struct PyModuleDef mymod_def = {
PyModuleDef_HEAD_INIT,
"mymod",
module_docstring,
-1,
module_methods};

//Initialize module
PyMODINIT_FUNC
PyInit_mymod(void)
{
PyObject *m = PyModule_Create(&mymod_def);
if (m == NULL)
return NULL;

//numpy functionallity
import_array();
return m;
}

Quite some code but overall not too complicated. We specified a docstring which will appear using help(mymod) etc. Then declared the functions that should go into the module, then specified the module methods and the module. Now we are free to start implementing the wrapper of the add function. 

static PyObject *add(PyObject *self, PyObject *args)
{
//PyObjects that should be parsed from args
PyObject *a_obj;
PyObject *b_obj;

//Check and parse..
if (!PyArg_ParseTuple(args, "OO", &a_obj, &b_obj))
return NULL;

//Numpy array from the parsed objects
//Yes you could check for type etc. but here we just convert to double
PyObject *a_array = PyArray_FROM_OTF(a_obj, NPY_DOUBLE, NPY_ARRAY_C_CONTIGUOUS);
PyObject *b_array = PyArray_FROM_OTF(b_obj, NPY_DOUBLE, NPY_ARRAY_C_CONTIGUOUS);
...
...
}

First we parse the arguments of the function to extract two Python object from which we later create numpy array objects. Since the data is stored continuous in memory we can then pass this to our C++ function. I won’t reiterate all the code here this is available on github. But after this we do some checks for size and shape of the array before getting the pointer to the data and calling cpp_add. 

    const double *a = (double *)PyArray_DATA((PyArrayObject *)a_array);
const double *b = (double *)PyArray_DATA((PyArrayObject *)b_array);

//And a pointer to the resutls
double *result = (double *)PyArray_DATA((PyArrayObject *)result_array);

//Now call add wih pointers and size
auto size = PyArray_Size(a_array);
cpp_add(a, b, result, size);

Final thing to do before returning is decreasing the reference count on the arrays we used. 

    Py_DECREF(a_array);
Py_DECREF(b_array);
return result_array;

For good measures I have also added continuous integration using Travis CI and tests using pytest. For details on that have a look at the code on https://github.com/erikfrojdh/python_cpp_example and if you have any questions please ask in the comments. Some other reading on the same topic can be found here: https://dfm.io/posts/python-c-extensions/

Next time we will see if there is an easier way to achieve the same result.

Leave a Reply

Your email address will not be published. Required fields are marked *