Parallelize iterative calculation using Theano

Question

This is a piece of python code that basically just calculates the activations of a neural network and then updates the new state for the next input value data[t] with respect to an arbitrary leaking rate a.

x = zeros((resSize,1))

for t in range(trainLen):
    u = data[t]

    xUpd = tanh( dot( Win, vstack((1,u)) ) + dot( W, x ) )
    x = (1-a) * x + a * xUpd 

    X[:,t] = vstack((1,u,x))[:,0]

It is not that important to understand what this is exactly doing. My question is: Can I parallelize this using the GPU with Theano? You can see that the new x depends on the previous value of x so what I would want to do is to parrallelize the calculations for those vectors and matrices. If those arrays become considerably large this will result in a much better performance.

Could anybody tell me how to do this?

Answer 1

This ticket was closed as off-topick, so I answered it in the comment. So here my answer as a post.

My answer is generic and apply to all system, not just Theano. As each iteration of your loop depend on the previous one, you can't paralelize your iterations completly. You could parallelize the u=data[t] as it don't depend on the previous x. You could parallelize dot( Win, vstack((1,u)) ) for the same reason. But you can't parallelize dot(W,x) and what depend on it like tanh and the lines afters.

If you want to optimize this, you can move outside the loop all computation that don't depend on x. This will allow to work with more data at the same time and so could be faster. So the dot(win, ...) could be speed up. But this will raise the memory usage.