boost python, using a namespace other than main global

Question 1

Before providing a solution, I want to provide some clarification on Python behavior.

Boost.Python's object is essentially a higher-level handle of a smart pointer. Thus, multiple object instances may point to the same Python object.

object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");

The above code imports a module named __main__. In Python, modules are essentially singletons due to the import behavior. Therefore, although the C++ main_module may be a member of the C++ PyExpression class, they all point to the same Python __main__ module, as it is a singleton. This results in main_namespace pointing to the same namespace.

Much of Python is built around dictionaries. For example, with an example module:

class Foo:
    def __init__(self):
        self.x = 42;

    def bar(self):
        pass

There are 3 dictionaries of interests:

example.__dict__ is the example module's namespace.

>>> example.__dict__.keys()
['__builtins__', '__file__', '__package__', '__name__', 'Foo', '__doc__']

example.Foo.__dict__ is a dictionary that describes the Foo class. Additionally, it would contain the equivalent of C++'s static member variables and functions.
```
>>> example.Foo.__dict__.keys()
['__module__', 'bar', '__doc__', '__init__']
```
example.Foo().__dict__ is a dictionary containing instance-specific variables. This would contain the equivalent of C++'s non-static member variables.
```
>>> example.Foo().__dict__.keys()
['x']
```

The Python exec statement takes two optional arguments:

The first argument specifies the dictionary that will be used for globals(). If the second argument is omitted, then it is also used for locals().
The second argument specifies the dictionary that will be used for locals(). Variable changes occurring within exec are applied to locals().

To get the desired behavior, example.Foo().__dict__ needs to be used as locals(). Unfortunately, this becomes slightly more complicated because of the following two factors:

Although import is a Python keyword, the CPython implementation is dependent on __builtins__.__import__. Thus, there needs to be a guarantee that the __builtin__ module is assessable as __builtins__ within the namespace passed to exec.
If a C++ class called Foo is exposed as a Python class through Boost.Python, then there is no easy way to access the Python Foo instance from within the C++ Foo instance.

To account for these behaviors, the C++ code will need to:

Get a handle to the Python object's __dict__.
Inject the __builtin__ module into the Python object's __dict__.
Extract the C++ object from the Python object.
Pass the Python object's __dict__ to the C++ object.

Here is an example solution that only sets variables on the instance for which code is being evaluated:

#include <boost/python.hpp>

class PyExpression
{
public:
  void run(boost::python::object dict) const
  {
    exec(exp_.c_str(), dict);
  }
  std::string exp_;
};

void PyExpression_run(boost::python::object self)
{
  // Get a handle to the Python object's __dict__.
  namespace python = boost::python;
  python::object self_dict = self.attr("__dict__");

  // Inject the __builtin__ module into the Python object's __dict__.
  self_dict["__builtins__"] = python::import("__builtin__");

  // Extract the C++ object from the Python object.
  PyExpression& py_expression = boost::python::extract<PyExpression&>(self);

  // Pass the Python object's `__dict__` to the C++ object.
  py_expression.run(self_dict);
}

BOOST_PYTHON_MODULE(PyExpModule)
{
  namespace python = boost::python;
  python::class_<PyExpression>("PyExpression")
    .def("run", &PyExpression_run)
    .add_property("exp", &PyExpression::exp_, &PyExpression::exp_)
    ;
}

// Helper function to check if an object has an attribute.
bool hasattr(const boost::python::object& obj,
             const std::string& name)
{
  return PyObject_HasAttrString(obj.ptr(), name.c_str());
}

int main()
{
  PyImport_AppendInittab("PyExpModule", &initPyExpModule);
  Py_Initialize();

  namespace python = boost::python;
  try
  {
    // python: import PyExpModule
    python::object py_exp_module = python::import("PyExpModule");

    // python: exp1 = PyExpModule.PyExpression()
    // python: exp1.exp = "import time; x = time.localtime().tm_year"
    python::object exp1 = py_exp_module.attr("PyExpression")();
    exp1.attr("exp") = 
      "import time;"
      "x = time.localtime().tm_year"
      ;

    // python: exp2 = PyExpModule.PyExpression()
    // python: exp2.exp = "import time; x = time.localtime().tm_mon"
    python::object exp2 = py_exp_module.attr("PyExpression")();
    exp2.attr("exp") = 
      "import time;"
      "x = time.localtime().tm_mon"
      ;

    // Verify neither exp1 nor exp2 has an x variable.
    assert(!hasattr(exp1, "x"));
    assert(!hasattr(exp2, "x"));

    // python: exp1.run()
    // python: exp2.run()
    exp1.attr("run")();
    exp2.attr("run")();

    // Verify exp1 and exp2 contain an x variable.
    assert(hasattr(exp1, "x"));
    assert(hasattr(exp2, "x"));

    // python: print exp1.x
    // python: print exp2.x
    std::cout << python::extract<int>(exp1.attr("x")) 
      << "\n" << python::extract<int>(exp2.attr("x"))
      << std::endl;
  }
  catch (python::error_already_set&)
  {
    PyErr_Print();
  }
}

And the output:

[twsansbury@localhost]$ ./a.out 
2013
5

Due to how libraries are loaded from imports, it may require providing arguments to the linker that will cause all symbols, not only used ones, to the dynamic symbol table. For example, when compiling the above example with gcc, using -rdynamic was required. Otherwise, import time will fail due to an undefined PyExc_IOError symbol.

Question 2

Python does not provide a 100% reliable isolation mechanism for this kind of task. That said, the essential tool you are looking for is the Python C-API Py_NewInterpreter, which is documented here. You will have to call it upon the creation of your PyExpression object, to create a new (semi)-isolated environment (N.B.: the destructor should call Py_EndInterpreter).

This is untested, but I'd guess something liket this would do the job:

PyThreadState* current_interpreter = Py_NewInterpreter();
bp::object pyrun = exec(expStr);
Py_EndInterpreter(current_interpreter);

You may wrap that into an object. If you wish to do so, you must manage the "thread" state as explained in this other stackoverflow thread.