Custom Class Serialization with JSON in Python 3

Question

I am having problems when JSON is trying to decode my serialized custom class with the class information encoded in the dict. I will try to provide as much information as possible, so please bear with the long post.

I am using the excellent tutorial provided here as reference. I am referring to the section which details writing encoding/decoding for your own classes.

My package-module structure is partially as follows:

+--- PyDev Project
    +-- src
        +-- astar
        |    +-- __init__.py
        |    +-- astar.py
        +-- common
        |    +-- __init__.py
        |    +-- service.py
        |    +-- graph.py
        |    +-- vertex.py

Each of these modules has its own class(es). I am serializing an object of the Service class in the common.service module. This is basically to use over a Pyro connection.

The serialization code is:

def convert_to_builtin_type(obj):
    print('default(', repr(obj), ')')
    # Convert objects to a dictionary of their representation
    d = { '__class__':obj.__class__.__name__, 
          '__module__':obj.__module__,
        }
d.update(obj.__dict__)
return d

The de-serialization code is:

def dict_to_object(self, d):
    if '__class__' in d:
        class_name = d.pop('__class__')
        module_name = d.pop('__module__')
        # module = __import__(module_name)
        module = import_module(module_name)
        print('MODULE:', module)
        class_ = getattr(module, class_name)
        print('CLASS:', class_)
        args = dict( (key.encode('ascii'), value) for key, value in d.items())
        print('INSTANCE ARGS:', args)
        inst = class_(**args)
    else:
        inst = d
    return inst

The problem is occurring during de-serialization. The local variable values are:

class_ = <module 'common.Service' from path-to-file.py>
class_name = 'Service'
module = <module 'common' from path-to-__init__.py>
module_name = 'common.Service'

The reason this is failing is because class_ is returning the package.module value and not the class name inside that module. Technically, class_ should hold common.service.Service which is the actual class. Due to this, the last statement inst = class(**args)_ is failing with a "TypeError: module object not callable" error.

I know that import or importlib.import_module both import that top level module, but in my case, how do I get to a class inside the second level module? Technically, the second level is the module and the first level is the package, so I need a class in the pkg.module that I am unable to get at.

I hope the question made sense and has enough research shown. I have more information if it needs to be clarified.

EDIT: Problem solved with User suggestion.

def dict_to_object(self, d):
    if '__class__' in d:
        class_name = d.pop('__class__')
        module_name = d.pop('__module__')
        # module = __import__(module_name)
        __import__(module_name)
        module = sys.modules[module_name]
        print('MODULE:', module)
        class_ = getattr(module, class_name)
        print('CLASS:', class_)
        args = dict((key, value) for key, value in d.items())
        print('INSTANCE ARGS:', args)
        inst = class_(**args)
    else:
        inst = d
    return inst

Answer 1

Ideas:

1. It could be that you problem is in this line:

   import_module(module_name)
   

   reconsider the arguments you pass to that function.

2. knowing the pickle serializer it does the following instead of import_module

   import sys
   __import__(module_name)
   module = sys.modules[module_name]
   

3. In Python 3 try using also obj.__class__.__qualname__. it includes the module name and the nesting classes.

   class X:
        class Y: pass # __qualname__ includes X.Y
   

4. I would be nice if you post the working code afterwards because this may be a problem for many people.