auto import all sub modules in a folder then invoke same name functions - python runtime inspect related

Question

I have an open folder on my computer,

OpenFunctions\
    ____Template.py
    function1.py
    function2.py
    ......
    functionx.py

This folder is experimental purpose for extend the ability of the whole app can doing. So let’s just think it is a quick and dry trying, no security consideration in this case.

My purpose is, if I drop a functionx.py following the ____Template.py, the app can know the new functions is available and will invoke the functions defined in this new joined file in someway – something like the plugin system, but should be a bit different.

So I wrote a ____inspect.py may let application have an ability to know what has been inputed.

Here is the

____inspect.py

def get_this_file_defined_functions(name_filter = "__"):
    import inspect, sys 
    f = inspect.getmembers(sys.modules[__name__], inspect.isfunction) 
    return [x for x in f if not x[0].startswith(name_filter)] 

def get_this_module_sub_modules(name_filter = "__"):
    import os.path, pkgutil
    pkgpath = os.path.dirname(__file__)
    m = [name for _, name, _ in pkgutil.iter_modules([pkgpath])] 
    return [x for x in m if not x[0].startswith(name_filter)]

def import_sub_modules_under_me(auto_exec_function = "auto_exec"):
    m = get_this_module_sub_modules()
    for i in m:  # need try except later
        exec "global %s; import %s" % (i, i)  
        #this will auto invoke __init__.py if sub modules folder is included
    for i in m:
        try:
            eval(i).eval(auto_exec_function)()
        except AttributeError:
            print "module %s has no function %s", % (i, auto_exec_function) 
        else:
            print "error on execute %s in module %s", % (auto_exec_function, i)

def execute_all_homonymy_functions(exec_function = "exec"):
    m = get_this_module_sub_modules()
    for i in m:
        #I need here for test if the module has been imported
        eval(i).eval(exec_function)()

Here is the

____Template.py

def __you_can_not_see_me(): pass  # because filtered by str.startswith()
def auto_exec(): pass             # this will be auto executed
def you_can_get_me(): pass
def you_can_get_me1(): pass
def you_can_get_me2(): pass

based on above idea I also want to extend structure to below

main.py
____inspect.py
OpenFunctions\
    __init__.py
    ____Template.py
    function1.py
    function2.py
    ......
    functionx.py
    module_aa
        \__init__.py
          aa.py
          aa1.py

Here is the main.py while the __init__.py may looks like

import ____inspect
____inspect.import_sub_modules_under_me()
____inspect.execute_all_homonymy_functions("what_ever_i_want")

Questions:

1. Above __init__ code will not working, because the sys.modules[__name__] is ____inspect when invoking but not the OpenFunctions or module_aa I want, is there a way to avoid pass the sys.modules[__name__] to the import_sub_modules_under_me() on the main.py or the __init__.py?

2. I suppose execute_all_homonymy_functions() will execute all the same name function in folder no matter it is exists in a sub module or in a single file, but I want to invoke all and the latest version in case the module new added or the source has been changed runtime. Then I want to use the code import aa, reload(aa) but may be thought as the wrong on below link, any suggestions? The issue I marked I need here for test if the module has been imported in __inspect.py

   [http://stackoverflow.com/questions/5027352/how-to-test-if-one-python-module-has-been-imported]

3. I also want to know the return type of one function in a file before invoking it, it was suggested to attach a decorate on each function. So my plan is:

\n

____decorate.py

def attrs(**kwds):
     def decorate(f):
         for k in kwds:
             setattr(f, k, kwds[k])
         return f
     return decorate

functionx.py
import ../____decorate.py

@attrs(argument_types=(int, int,),returns=int)
def __you_can_not_see_me(): pass

@attrs(argument_types=(int, int,),returns=int)
def auto_exec(): pass             # I will be auto executed

@attrs(argument_types=(int, int,),returns=int)
def you_can_get_me(): pass

@attrs(argument_types=(int, int,),returns=int)
def you_can_get_me1(): pass

@attrs(argument_types=(int, int,),returns=int)
def you_can_get_me2(): pass

Is it works ok for inspect case? Or there’s an better solution?

The last one: below code

exec "global %s; import %s" % (i, i) 
eval(i).eval(auto_exec_function)()

looks ugly, any alternative for above two lines?

Thanks for your help.

Rgs, KC

Answer 1

To address your last question first: to dynamically import modules in recent versions of Python, use importlib.import_module(), that's what it is for. If you're on an older version of Python, you should use __import__ instead (but check the idiosyncracies of that direct approach in the docs - there's a reason the import_module() function was added as a replacement).

To address your first question: there is no officially portable way to retrieve information about the calling function's global environment, so the most correct answer is to simply pass __name__ in as an argument wherever it is needed.

Reloading in Python is always a little dangerous, since not everything works correctly under reloading (even many standard library modules will fail if you reload either them, or a module they reference).

I really suggest spending some time exploring the answers to this existing question about plugin architectures in Python: Building a minimal plugin architecture in Python