Make a dynamic import which is still explicit in Python

Question

Okay, this is a bit complex.

Let's say I have a module inside a package:

a_package
 |-- __init__.py
 |-- a_module.py

Inside a_module.py I declare A_Class:

# file location: a_package/a_module.py
class A_Class():
    def say(self):
        print ("cheese")

I can make an instance of A_Class and call say method by doing this:

from a_package.a_module import A_Class
my_object = A_Class()
my_object.say() # this will display 'cheese' as expected

However, I want to make a more dynamic approach (I plan to have a lot of packages and classes, and want to make the code easier to write). So, I make a function called load_class

def load_class(package_name, module_name, class_name)
    result = None
    try:
        exec('from ' + package_name + '.' + module_name + ' import ' + class_name)
        exec('result = '+class_name)
    except:
        raise ImportError('Unable to load the class')
    return result

# Now, I can conveniently do this:
A_Class = load_class('a_package', 'a_module', 'A_Class')
my_object = A_Class()
my_object.say()

# or even shorter:
load_class('a_package', 'a_module', 'A_Class')().say()

The program works as expected, but the IDE (I use pydev) doesn't understand my code, and cannot do the intellisense (automatically completing the code).

If I use the first approach, the intellisense is obviously works:

from a_package.a_module import A_Class
my_object = A_Class()
my_object. # when I press ".", there will be a popup to let me choose "say" method

But if I use the second approach, the intellisense cannot do the completion for me:

load_class('a_package', 'a_module', 'A_Class')(). # when I press ".", nothing happened

I know, this is a trade-off of do dynamic import in Python. But, I want to know if there is some alternative to let me do the second approach (may be not using exec) that can still let the intellisense of generic IDE (such as Pydev) guess the method inside the class?

EDIT: Why I need to do this? Let's say I have such a directory structure

fruit
 |-- strawberry.py
 |-- orange.py

chocolate
 |-- cadbury.py
 |-- kitkat.py

need_dynamic.py

And in need_dynamic.py, I have this script:

food_list = ['fruit', 'chocolate']
subfood_list = [['strawberry', 'orange'],['cadbury', 'kitkat']] 
# show food list and ask user to choose food
for i in xrange(len(food_list)):
    print i + " : " + food_list[i]
food_index = int(raw_input('chose one'))
# show subfood list and ask user to choose subfood 
for i in xrange(len(subfood_list[food_index])):
    print i + " : " + subfood_list[food_index][i]
subfood_index = int(raw_input('chose one'))
# init the class
my_class = load_class(food_list[food_index], subfood_list[food_index, subfood_index])
# the rest of the code

This is just for simplification, actually, I plan to fill food_list and subfood_list automatically by fetch the directory.

Imagine you have a data-classification framework, and want to let a user choose what method they want to use. The user should also be able to extend the framework by simply adding python package an module.

I hope this example is reasonable.

edit again the accepted answer doesn't solve the intellisense problem. But it shows how to code better. I think it is IDE problem rather than python problem. I 'll post another question.

Answer 1

Ok here is your solution :

For Importing the module using name as string:

__import__(modulename, globals(), locals(), ['*'])

Load the class from the module path:

cls = getattr(sys.modules[modulename], classname)

Directory structure:

:/tmp/dynamic_import:~ ls
chocolate   fruit       need_dynamic.py
:/tmp/dynamic_import:~

need_dynamic.py

fruit
  |-- strawberry.py
  |-- orange.py

chocolate
 |-- cadbury.py
 |-- kitkat.py

Here is one of the module inside fruit, i name the class name with respect to module name with initials caps:

:/tmp/dynamic_import:~ cat orange.py
class Orange(object):
    def say(self):
        return "Say cheese from class: %s" % __name__
:/tmp/dynamic_import:~

Here is your main script:

#!/usr/bin/env python

import os
import sys
import inspect

def load_modules_from_path(path):
    """
    Import all modules from the given directory
    """
    # Check and fix the path
    if path[-1:] != '/':
        path += '/'

    # Get a list of files in the directory, if the directory exists
    if not os.path.exists(path):
         raise OSError("Directory does not exist: %s" % path)

    # Add path to the system path
    sys.path.append(path)
    # Load all the files in path
    for f in os.listdir(path):
        # Ignore anything that isn't a .py file
        if len(f) > 3 and f[-3:] == '.py':
            modname = f[:-3]
            # Import the module
            __import__(modname, globals(), locals(), ['*'])

def load_class_from_name(fqcn):

    # fqcn = fully qualified classname
    # Break apart fqcn to get module and classname

    paths = fqcn.split('.')
    modulename = '.'.join(paths[:-1])
    classname = paths[-1]

    # Import the module
    __import__(modulename, globals(), locals(), ['*'])

    # Get the class
    cls = getattr(sys.modules[modulename], classname)
    # Check cls
    if not inspect.isclass(cls):
        raise TypeError("%s is not a class" % fqcn)

    # Return class
    return cls

def main():

    food_list = ['fruit', 'chocolate']
    subfood_list = [['strawberry', 'orange'],['cadbury', 'kitkat']]

    # show food list for users to select.
    for i in xrange(len(food_list)):
        print '%d: %s' % (i, food_list[i])

    food_index = int(raw_input('Choose one: '))

    for i in xrange(len(subfood_list[food_index])):
        print '%d: %s' % (i, subfood_list[food_index][i])

    subfood_index = int(raw_input('Chose one: '))
    pkg = food_list[food_index]
    module = subfood_list[food_index][subfood_index]
    class_name = module.title()

    load_modules_from_path(pkg)
    new_class = load_class_from_name('%s.%s' % (module, class_name))

    # instantiation
    obj = new_class()
    print obj.say()

if __name__ == '__main__': main()

Here is the output:

:/tmp/dynamic_import:~ python need_dynamic.py
0: fruit
1: chocolate
Choose one: 0
0: strawberry
1: orange
Chose one: 0
Say cheese from class: strawberry
:/tmp/dynamic_import:~ python need_dynamic.py
0: fruit
1: chocolate
Choose one: 1
0: cadbury
1: kitkat
Chose one: 0
Say cheese from class: cadbury
:/tmp/dynamic_import:~

Please let me know if that works.

Answer 2

You'd want to use the __import__ builtin:

def load_class(package, mod_name, cls_name):
    mod = __import__('.'.join((package, mod_name)))
    return getattr(mod, cls_name)

Of course, you can throw your error handling back in there if you'd like, and honestly I'm not really sure why you'd want to do this in the first place. Dynamic importing seems like a "code smell" to me for most things. Please evaluate whether you actually need this before you go off and start using it.