Python: How to import all methods and attributes from a module dynamically

Question

I'd like to load a module dynamically, given its string name (from an environment variable). I'm using Python 2.7. I know I can do something like:

import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))

This is roughly equivalent to

import my_settings

(where SETTINGS_MODULE = 'my_settings'). The problem is, I need something equivalent to

from my_settings import *

since I'd like to be able to access all methods and variables in the module. I've tried

import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
from my_module import *

but I get a bunch of errors doing that. Is there a way to import all methods and attributes of a module dynamically in Python 2.7?

Answer 1

If you have your module object, you can mimic the logic import * uses as follows:

module_dict = my_module.__dict__
try:
    to_import = my_module.__all__
except AttributeError:
    to_import = [name for name in module_dict if not name.startswith('_')]
globals().update({name: module_dict[name] for name in to_import})

However, this is almost certainly a really bad idea. You will unceremoniously stomp on any existing variables with the same names. This is bad enough when you do from blah import * normally, but when you do it dynamically there is even more uncertainty about what names might collide. You are better off just importing my_module and then accessing what you need from it using regular attribute access (e.g., my_module.someAttr), or getattr if you need to access its attributes dynamically.

Answer 2

Not answering precisely the question as worded, but if you wish to have a file as proxy to a dynamic module, you can use the ability to define __getattr__ on the module level.

import importlib
import os

module_name = os.environ.get('CONFIG_MODULE', 'configs.config_local')
mod = importlib.import_module(module_name)

def __getattr__(name):
    return getattr(mod, name)

Answer 3

My case was a bit different - wanted to dynamically import the constants.py names in each gameX.__init__.py module (see below), cause statically importing those would leave them in sys.modules forever (see: this excerpt from Beazley I picked from this related question).

Here is my folder structure:

game/
 __init__.py
 game1/
   __init__.py
   constants.py
   ...
 game2/
   __init__.py
   constants.py
   ...

Each gameX.__init__.py exports an init() method - so I had initially a from .constants import * in all those gameX.__init__.py which I tried to move inside the init() method.

My first attempt in the lines of:

@@ -275,2 +274,6 @@ def init():
     # called instead of 'reload'
+    yak = {}
+    yak.update(locals())
+    from .constants import * # fails here
+    yak = {x: y for x,y in locals() if x not in yak}
+    globals().update(yak)
     brec.ModReader.recHeader = RecordHeader

Failed with the rather cryptic:

SyntaxError: import * is not allowed in function 'init' because it contains a nested function with free variables

I can assure you there are no nested functions in there. Anyway I hacked and slashed and ended up with:

def init():
    # ...
    from .. import dynamic_import_hack
    dynamic_import_hack(__name__)

Where in game.__init__.py:

def dynamic_import_hack(package_name):
    print __name__ # game.init
    print package_name # game.gameX.init
    import importlib
    constants = importlib.import_module('.constants', package=package_name)
    import sys
    for k in dir(constants):
        if k.startswith('_'): continue
        setattr(sys.modules[package_name], k, getattr(constants, k))

(for setattr see How can I add attributes to a module at run time? while for getattr How can I import a python module function dynamically? - I prefer to use those than directly access the __dict__)

This works and it's more general than the approach in the accepted answer cause it allows you to have the hack in one place and use it from whatever module. However I am not really sure it's the best way to implement it - was going to ask a question but as it would be a duplicate of this one I am posting it as an answer and hope to get some feedback. My questions would be:

• why this "SyntaxError: import * is not allowed in function 'init'" while there are no nested functions ?
• dir has a lot of warnings in its doc - in particular it attempts to produce the most relevant, rather than complete, information - this complete worries me a bit
• is there no builtin way to do an import * ? even in python 3 ?