Getting respective *args, **kwargs from collection of functions, then filling in the irregular 2d structure with provided 1d arguments

Question

I have data like this:

args, kwargs = (('foo', 'bar', 'baz'), {'goo': 1})

And I have functions, inside an object, which want these data as arguments. They are to be provided via a method which has this sort of signature (represented just as *args):

callFunctions((*args, **kwargs), ...)

Or, more explicitly in its structure:

callFunctions(((args ...), {kwargs ...}), ...)

(I hope it's clear enough what I'm expecting in this method.)

Let's say, for the sake of example, that my two functions are as follows:

def func1(foo, bar):
    print foo, bar

def func2(baz, goo=0):
    print baz, goo

funcs = func1, func2    # for iteration

With that background, here are the two problems I'm having.

Getting the format of *args, **kwargs for each function

I have been trying to use the inspect module to get the argument spec for each function, so that I may 'fill in' the structure with the 1d data in args and kwargs. I've tried the following:

format = [(spec.args, spec.keywords) for spec in (inspect.getargspec(func) for func in funcs)]

But I don't understand why the spec.keywords always gives me None (seems kind of asinine). The default value of any keyword argument will appear in spec.defaults, but I don't know how to associate it with the correct keyword argument. (It's also frustrating that all of the keyword arguments are put into spec.args.)

[(['foo', 'bar'], None), (['baz', 'goo'], None)]

Filling in the structure to be passed to callFunctions

Assuming I had the structure anyway, filling that structure in with the original data is tricky: Basically, I want to fill the first tuple of *args, **kwargs with the first however-many positional arguments, and the appropriate keyword arguments; then, move on to the next tuple with the next however-many positional arguments and appropriate keyword arguments; and so on. I've tried the following:

argues = []
positional_index = 0
format = ((('foo', 'bar'), {}), (('baz',), {'goo': 0}))
for pair in format:
    however_many = len(pair[0])
    argues.append((tuple(args[positional_index:however_many]), dict({(k, kwargs[k]) for k in pair[1]})))
    positional_index += however_many

But I get this:

[(('foo', 'bar'), {}), ((), {'goo': 1})]

Why don't I get baz?

Answer 1

For the first part of your question: The thing is that (in Python 2) there isn't a complete distinction between "keyword" and "positional" arguments as such. Any argument value can be passed positionally or by keyword. It's just that if the argument doesn't have a default value, that argument must be supplied --- but it can still be passed by keyword. (In Python 3 it's possible to have true keyword-only arguments that cannot be passed positionally.)

The way to match up the arguments with their defaults, then, is to realize that the defaults can only apply to the last arguments. So if a function accepts four arguments and has two default argument values, those values must be for the third and fourth arguments, with the first two arguments having no default values. In common parlance, this is a function with two positional arguments and two keyword arguments (although, as I mentioned, this is only half-accurate, since any argument can always be passed by keyword). As the documentation says, the keywords part of the result of getargspec is not meant to hold the keyword arguments; it holds the name of the keyword varargs argument (e.g., **kwargs), if there is one.

To see how to match them up, just look at the source code for inspect.formatargspec, which does essentially this (I changed it slightly to build a list instead of a string representation):

args, varargs, varkw, defaults = inspect.getargspec(func)
result = []
if defaults:
    firstdefault = len(args) - len(defaults)
for i, arg in enumerate(args):
    if defaults and i >= firstdefault:
        result.append((arg, defaults[i - firstdefault]))
    else:
        result.append((arg,))

For the second part of your question, the problem is just that you're slicing from positional_index:however_many instead of positional_index:positional_index+however_many. Also, your dict/set confusion is because you're using a set comprehension instead of a dict comprehension. Plus, you don't need that call to tuple. Do it like this:

for pair in format:
    however_many = len(pair[0])
    argues.append((args[positional_index:positional_index+however_many], {k: kwargs[k] for k in pair[1]}))
    positional_index += however_many

Note, though, that one problem with this is you'll be unable to pass different-valued keyword arguments of the same name to different functions. If you have def foo(a, b, c=2) and def bar(a, b, c=8), you have no way to pass {'c': 1} to foo and {'c': 2} to bar, because you're only passing one dict at the beginning, which can only have one 'c' key.

More generally, if you really want to handle any legal argument set, it won't be as simple as this, because even "positional" arguments can be passed by keyword. If you have a function like def foo(a, b, c=3, d=4), it's legal to call it with foo(1, d=8, c=7, b=6) --- you can pass the keyword arguments out of order, and also pass a value for b by keyword, even though it has no default value. So you can't in general just grab positional arguments based on the number of arguments without default values; you have to actually look at which particular arguments were passed by keyword, and only positionally pass the ones that weren't passed by keyword. (You could of course just say that your function won't work for this sort of case; it depends how general you want it to be.)