Most Pythonic way to provide function metadata at compile time?

Question

I am building a very basic platform in the form of a Python 2.7 module. This module has a read-eval-print loop where entered user commands are mapped to function calls. Since I am trying to make it easy to build plugin modules for my platform, the function calls will be from my Main module to an arbitrary plugin module. I'd like a plugin builder to be able to specify the command that he wants to trigger his function, so I've been looking for a Pythonic way to remotely enter a mapping in the command->function dict in the Main module from the plugin module.

I've looked at several things:

1. Method name parsing: the Main module would import the plugin module and scan it for method names that match a certain format. For example, it might add the download_file_command(file) method to its dict as "download file" -> download_file_command. However, getting a concise, easy-to-type command name (say, "dl") requires that the function's name also be short, which isn't good for code readability. It also requires the plugin developer to conform to a precise naming format.

2. Cross-module decorators: decorators would let the plugin developer name his function whatever he wants and simply add something like @Main.register("dl"), but they would necessarily require that I both modify another module's namespace and keep global state in the Main module. I understand this is very bad.

3. Same-module decorators: using the same logic as above, I could add a decorator that adds the function's name to some command name->function mapping local to the plugin module and retrieve the mapping to the Main module with an API call. This requires that certain methods always be present or inherited though, and - if my understanding of decorators is correct - the function will only register itself the first time it is run and will unnecessarily re-register itself every subsequent time thereafter.

Thus, what I really need is a Pythonic way to annotate a function with the command name that should trigger it, and that way can't be the function's name. I need to be able to extract the command name->function mapping when I import the module, and any less work on the plugin developer's side is a big plus.

Thanks for the help, and my apologies if there are any flaws in my Python understanding; I'm relatively new to the language.

Answer 1

User-defined functions can have arbitrary attributes. So you could specify that plug-in functions have an attribute with a certain name. For example:

def a():
  return 1

a.command_name = 'get_one'

Then, in your module you could build a mapping like this:

import inspect #from standard library

import plugin

mapping = {}

for v in plugin.__dict__.itervalues():
    if inspect.isfunction(v) and v.hasattr('command_name'):
        mapping[v.command_name] = v

To read about arbitrary attributes for user-defined functions see the docs

Answer 2

Building or Standing on the first part of @ericstalbot's answer, you might find it convenient to use a decorator like the following.

################################################################################
import functools
def register(command_name):
    def wrapped(fn):
        @functools.wraps(fn)
        def wrapped_f(*args, **kwargs):
            return fn(*args, **kwargs)
        wrapped_f.__doc__ += "(command=%s)" % command_name
        wrapped_f.command_name = command_name
        return wrapped_f
    return wrapped
################################################################################
@register('cp')
def copy_all_the_files(*args, **kwargs):
    """Copy many files."""
    print "copy_all_the_files:", args, kwargs
################################################################################

print  "Command Name: ", copy_all_the_files.command_name
print  "Docstring   : ", copy_all_the_files.__doc__

copy_all_the_files("a", "b", keep=True)

Output when run:

Command Name:  cp
Docstring   :  Copy many files.(command=cp)
copy_all_the_files: ('a', 'b') {'keep': True}

Answer 3

There are two parts in a plugin system:

1. Discover plugins
2. Trigger some code execution in a plugin

The proposed solutions in your question address only the second part.

There many ways to implement both depending on your requirements e.g., to enable plugins, they could be specified in a configuration file for your application:

plugins = some_package.plugin_for_your_app
    another_plugin_module
    # ...

To implement loading of the plugin modules:

plugins = [importlib.import_module(name) for name in config.get("plugins")]

To get a dictionary: command name -> function:

commands = {name: func 
            for plugin in plugins
            for name, func in plugin.get_commands().items()}

Plugin author can use any method to implement get_commands() e.g., using prefixes or decorators — your main application shouldn't care as long as get_commands() returns the command dictionary for each plugin.

For example, some_plugin.py (full source):

def f(a, b):
    return a + b

def get_commands():
    return {"add": f, "multiply": lambda x,y: x*y}

It defines two commands add, multiply.