Can I pass the objective and derivative functions to scipy.optimize.minimize as one function?

Question

I'm trying to use scipy.optimize.minimize to minimize a complicated function. I noticed in hindsight that the minimize function takes the objective and derivative functions as separate arguments. Unfortunately, I've already defined a function which returns the objective function value and first-derivative values together -- because the two are computed simultaneously in a for loop. I don't think there is a good way to separate my function into two without the program essentially running the same for loop twice.

Is there a way to pass this combined function to minimize?

(FYI, I'm writing an artificial neural network backpropagation algorithm, so the for loop is used to loop over training data. The objective and derivatives are accumulated concurrently.)

Answer 1

Something that might work is: you can memoize the function, meaning that if it gets called with the same inputs a second time, it will simply return the same outputs corresponding to those inputs without doing any actual work the second time. What is happening behind the scenes is that the results are getting cached. In the context of a nonlinear program, there could be thousands of calls which implies a large cache. Often with memoizers(?), you can specify a cache limit and the population will be managed FIFO. IOW you still benefit fully for your particular case because the inputs will be the same only when you are needing to return function value and derivative around the same point in time. So what I'm getting at is that a small cache should suffice.

You don't say whether you are using py2 or py3. In Py 3.2+, you can use functools.lru_cache as a decorator to provide this memoization. Then, you write your code like this:

@functools.lru_cache
def original_fn(x):
   blah
   return fnvalue, fnderiv

def new_fn_value(x):
   fnvalue, fnderiv = original_fn(x)
   return fnvalue

def new_fn_deriv(x):
   fnvalue, fnderiv = original_fn(x)
   return fnderiv

Then you pass each of the new functions to minimize. You still have a penalty because of the second call, but it will do no work if x is unchanged. You will need to research what unchanged means in the context of floating point numbers, particularly since the change in x will fall away as the minimization begins to converge.

There are lots of recipes for memoization in py2.x if you look around a bit.

Did I make any sense at all?

Answer 2

Yes, you can pass them in a single function:

import numpy as np
from scipy.optimize import minimize

def f(x):
    return np.sin(x) + x**2, np.cos(x) + 2*x

sol = minimize(f, [0], jac=True, method='L-BFGS-B')