Automatic conversion to boost shared_ptr in boost python

Question

I register a converter that converts a 2 dimensional NumPy-Array to an Eigen Matrix (called DenseMatrix):

boost::python::converter::registry::push_back(
        &convertible,
        &construct,
        boost::python::type_id<DenseMatrix>()
);

If I have a function accepting a DenseMatrix-object and export it to Python I can call it from Python with a NumPy-Array. On the other hand, if I have a function accepting a boost::shared_ptr (or a pointer or reference) and export it to Python I get an error like this when calling it from Python:

Boost.Python.ArgumentError: Python argument types in
    libNSM.takeReference(numpy.ndarray)
did not match C++ signature:
    takeReference(Eigen::Matrix<double, -1, -1, 1, -1, -1> {lvalue})

I don't want to write an explicit converter from a NumPy-Array to a boost::shared_ptr. Is there a better way to go about this?

Here is the corresponding code, just to be clear:

void takeObject(DenseMatrix m) {
    // ...
}
void takePointer(DenseMatrix* ptr) {
    // ...
}
void takeReference(DenseMatrix* ptr) {
    // ...
}
void takeSharedPointer(boost::shared_ptr<DenseMatrix> ptr) {
    // ...
}
BOOST_PYTHON_MODULE(boostModule) {
    def("takeObject", &takeObject);
    def("takePointer", &takePointer);
    def("takeReference", &takeReference);
    def("takeSharedPointer", &takeSharedPointer);
}

Answer 1

If you need a modifiable DenseMatrix and making copies of DenseMatrix is expensive, then having the converter that converts a numpy.ndarray to a copy constructable smart pointer that manages a DenseMatrix, such as boost::shared_ptr, may be the best option.

Boost.Python does not support custom lvalue converters. To reduce the chance of a dangling reference and provide explicit directionality between the languages, Boost.Python will pass the object resulting from the conversion by const reference to functions. Therefore, when a custom converter is providing an argument, the functions parameter must either accept the argument by value or by a const reference:

• takeObject() works as the DenseMatrix parameter will be constructed via a copy constructor.
• takeReference(DenseMatrix&) fails as DenseMatrix& cannot bind to const DenseMatrix&.

If a custom converter is already being registered that converts a numpy.ndarray to a DenseMatrix, then it may be possible to reuse much of the code when registering a numpy.ndarray to a smart pointer that manages DenseMatrix.

Here is a complete example showing code being shared between two custom converters that convert Python objects contains an int x and y attribute to a Spam object. Additionally, the example shows how auxiliary functions can be used to pass converted objects by a non-const reference or pointer.

#include <iostream>
#include <boost/make_shared.hpp>
#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>

/// @brief Mockup Spam class.
struct Spam
{
  int x;
  int y;
  Spam()  { std::cout << "Spam()" << std::endl; }
  ~Spam() { std::cout << "~Spam()" << std::endl; }

  Spam(const Spam& rhs) : x(rhs.x), y(rhs.y)
    { std::cout << "Spam(const Spam&)" << std::endl; }
};

/// @brief Helper function to ceck if an object has an attributed with a
///        specific type.
template <typename T>
bool hasattr(const boost::python::object& obj,
             const char* name)
{
  return PyObject_HasAttrString(obj.ptr(), name) &&
         boost::python::extract<T>(obj.attr(name)).check();
}

/// @brief Helper type that provides conversions from a Python object to Spam.
struct spam_from_python
{
  spam_from_python()
  {
    boost::python::converter::registry::push_back(
      &spam_from_python::convertible,
      &spam_from_python::construct,
      boost::python::type_id<Spam>());
  }

  /// @brief Check if PyObject contains an x and y int attribute.
  static void* convertible(PyObject* object)
  {
    namespace python = boost::python;
    python::handle<> handle(python::borrowed(object));
    python::object o(handle);

    // If x and y are not int attributes, then return null.
    if (!hasattr<int>(o, "x") && hasattr<int>(o, "y"))
      return NULL;

    return object;
  }

  /// @brief Convert PyObject to Spam.
  static void construct(
    PyObject* object,
    boost::python::converter::rvalue_from_python_stage1_data* data)
  {
    // Obtain a handle to the memory block that the converter has allocated
    // for the C++ type.
    namespace python = boost::python;
    typedef python::converter::rvalue_from_python_storage<Spam> storage_type;
    void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

    // Allocate the C++ type into the converter's memory block, and assign
    // its handle to the converter's convertible variable.
    Spam* spam;
    data->convertible = spam = new (storage) Spam();

    // Initialize spam from an object.
    initialize_spam(spam, object);
  }

  /// @brief Initialize a spam instance based on a python object.
  static void initialize_spam(Spam* spam, PyObject* object)
  {
    namespace python = boost::python;
    python::handle<> handle(python::borrowed(object));
    python::object o(handle);

    spam->x = python::extract<int>(o.attr("x"));
    spam->y = python::extract<int>(o.attr("y"));
  } 
};

/// @brief Helper type that provides conversions from a Python object to
///        boost::shared_ptr<Spam>.
struct shared_spam_from_python
{
  shared_spam_from_python()
  {
    boost::python::converter::registry::push_back(
      &spam_from_python::convertible,
      &shared_spam_from_python::construct,
      boost::python::type_id<boost::shared_ptr<Spam> >());
  }

  /// @brief Convert PyObject to boost::shared<Spam>.
  static void construct(
    PyObject* object,
    boost::python::converter::rvalue_from_python_stage1_data* data)
  {
    // Obtain a handle to the memory block that the converter has allocated
    // for the C++ type.
    namespace python = boost::python;
    typedef python::converter::rvalue_from_python_storage<
                                        boost::shared_ptr<Spam> > storage_type;
    void* storage = reinterpret_cast<storage_type*>(data)->storage.bytes;

    // Allocate the C++ type into the converter's memory block, and assign
    // its handle to the converter's convertible variable.
    boost::shared_ptr<Spam>* spam;
    data->convertible = spam =
        new (storage) boost::shared_ptr<Spam>(boost::make_shared<Spam>());

    // Initialize spam from an object.
    spam_from_python::initialize_spam(spam->get(), object);
  }
};

/// @brief Mockup functions acceping Spam in different ways.
void by_value(Spam spam)            { std::cout << "by_value()" << std::endl; }
void by_const_ref(const Spam& spam) { std::cout << "by_cref()"  << std::endl; }
void by_ref(Spam& spam)             { std::cout << "by_ref()"   << std::endl; }
void by_ptr(Spam* spam)             { std::cout << "by_ptr()"   << std::endl; }

/// @brief Use auxiliary functions that accept boost::shared_ptr<Spam> and 
///        delegate to functions that have formal parameters of Spam& and
///        Spam*.
void by_ref_wrap(boost::shared_ptr<Spam> spam) { return by_ref(*spam); }
void by_ptr_wrap(boost::shared_ptr<Spam> spam) { return by_ptr(spam.get()); }

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // Enable python to Spam conversion.
  spam_from_python();

  // Enable python to boost::shared_ptr<Spam> conversion.
  shared_spam_from_python();

  // Expose functions that have parameters that can accept a const Spam&
  // argument.
  python::def("by_value",     &by_value);
  python::def("by_const_ref", &by_const_ref);

  // Expose functions that have parameters that can accept a const
  // boost::shared_ptr<Spam>& argument.  As copies of shared_ptr are cheap,
  // a copy is used and the managed instance is passed to other functions,
  // allowing Spam& and Spam* parameters.
  python::def("by_ptr", &by_ptr_wrap);
  python::def("by_ref", &by_ref_wrap);
}

Interactive usage:

>>> class Egg:
...     x = 1
...     y = 2
... 
>>> import example
>>> example.by_value(Egg())
Spam()
Spam(const Spam&)
by_value()
~Spam()
~Spam()
>>> example.by_const_ref(Egg())
Spam()
by_cref()
~Spam()
>>> example.by_ref(Egg())
Spam()
by_ref()
~Spam()
>>> example.by_ptr(Egg())
Spam()
by_ptr()
~Spam()