Numerically Stable Implementation

Question

I need to compute a normalized exponential of a vector in Matlab.

Simply writing

res = exp(V)/sum(exp(V))

overflows in an element of V is greater than log(realmax) = 709.7827. (I am not sure about underflow conditions.)

How should I implement it to avoid numerical instability?

Update: I received excellent responses about how to avoid overflow. However, I am still happy to hear your thoughts about the possibility of underflow in the code.

Answer 1

The following approach avoids the overflow by subtracting the exponents and then taking the exponential, instead of dividing the exponentials:

res = 1./sum(exp(bsxfun(@minus, V(:), V(:).')))

As a general rule, overflow can be avoided by working in the log domain for as long as possible, and taking the exponential only at the end.

Answer 2

The answer is pretty similar to your previous question. Use Math!

exp(V)=exp(V-max(V))*exp(max(V))
sum(exp(V))=sum(exp(V-max(V))*exp(max(V)))=exp(max(V)*sum(exp(V-max(V))))

Putting both together:

res=exp(V-max(V))*exp(max(V))/exp(max(V)*sum(exp(V-max(V)))=exp(V-max(V))/sum(exp(V-max(V)))

A code which is robust to the input range:

res=exp(V-max(V))/sum(exp(V-max(V)))