Numpy error: shape mismatch

Question

When I was trying to solve a scientific problem with Python (Numpy), a 'shape mismatch' error came up: "shape mismatch: objects cannot be broadcast to a single shape". I managed to reproduce the same error in a simpler form, as shown below:

import numpy as np
nx = 3; ny = 5
ff = np.ones([nx,ny,7])
def test(x, y):
    z = 0.0
    for i in range(7):
        z = z + ff[x,y,i]
    return z

print test(np.arange(nx),np.arange(ny))

When I tried to call test(x,y) with x=1,y=np.arange(ny), everything works fine. So what's going on here? Why can't the both parameters be numpy arrays?

UPDATE

I have worked out the problem with some hints from @Saullo Castro. Here's some updated info for you guys who tried to help but feel unclear about my intention:

Basically I created a mesh grid with dimension nx*ny and another array ff that stores some value for each node. In the above code, ff has 7 values for each node and I was trying to sum up the 7 values to get a new nx*ny array.

However, the "shape mismatch" error is not due to the summing process as many of you might have guess now. I have misunderstood the rule of functions taking ndarray objects as input parameters. I tried to pass np.arange(nx), np.arange(ny) to test() is not gonna give me what I desired, even if nx==ny.

Back to my original intention, I solve the problem by creating another function and used np.fromfunction to created the array:

def tt(x, y):  
    return np.fromfunction(lambda a,b: test(a,b), (x, y))

which is not perfect but it works. (In this example there seems to be no need to create a new function, but in my actual code I modified it a bit so it can be used for slice of the grid)

Anyway, I do believe there's a much better way compared to my kind of dirty solution. So if you have any idea about that, please share with us :).

Answer 1

Let's look into an array similar to your ff array:

nx = 3; ny = 4
ff = np.arange(nx*ny*5).reshape(nx,ny,5)
#array([[[ 0,  1,  2,  3,  4],
#        [ 5,  6,  7,  8,  9],
#        [10, 11, 12, 13, 14],
#        [15, 16, 17, 18, 19]],
#
#       [[20, 21, 22, 23, 24],
#        [25, 26, 27, 28, 29],
#        [30, 31, 32, 33, 34],
#        [35, 36, 37, 38, 39]],
#
#       [[40, 41, 42, 43, 44],
#        [45, 46, 47, 48, 49],
#        [50, 51, 52, 53, 54],
#        [55, 56, 57, 58, 59]]])

When you index using arrays of indices a, b, c like in ff[a, b, c], a, b, c must have the same shape, and numpy will build a new array based on the indices. For example:

ff[[0, 0, 1, 1, 2, 2], [0, 1, 0, 1, 2, 3], [0, 0, 0, 1, 1, 1]]
#array([ 0,  5, 20, 26, 51, 56])

This is called fancy indexing, which is like building an array with:

np.array([ff[0, 0, 0], ff[0, 1, 0], ff[1, 0, 0], ..., ff[2, 3, 1]])

In your case the f[x, y, i] will produce a shape mismatch error since a, b, c do not have the same shape.

Answer 2

Looks like you want to sum ff over the last dimension, with the 1st 2 dimensions covering their whole range. : is used to denote the whole range of a dimension:

def test():
    z = 0.0
    for i in range(7):
        z = z + ff[:,:,i]
    return z
print test()

But you can get the same result without looping, by using the sum method.

print ff.sum(axis=-1)

: is shorthand for 0:n

ff[0:nx, 0:ny, 0]==ff[:,:,0]

It is possible to index a block of ff with ranges, but you have to be much more careful about the shapes of the indexing arrays. For a beginner it is better to focus on getting slicing and broadcasting correct.

---

edit -

You can index an array like ff with arrays generated by meshgrid:

I,J = meshgrid(np.arange(nx),np.arange(ny),indexing='ij',sparse=False)
I.shape # (nx,ny)
ff[I,J,:]

also works with

I,J = meshgrid(np.arange(nx),np.arange(ny),indexing='ij',sparse=True)
I.shape # (nx,1)
J.shape # (1, ny)

ogrid and mgrid are alternatives to meshgrid.

Answer 3

Let's reproduce your problem in 2D case, so it is easier to see:

import numpy as np

a = np.arange(15).reshape(3,5)

x = np.arange(3)
y = np.arange(5)

Demo:

>>> a
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])
>>> a[x, y] # <- This is the error that you are getting
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: shape mismatch: objects cannot be broadcast to a single shape

# You are getting the error because x and y are different lengths,
# If x and y were the same lengths, the code would work: 

>>> a[x, x]
array([ 0,  6, 12])

# mixing arrays and scalars is not a problem

>>> a[x, 2]
array([ 2,  7, 12])

Answer 4

It is not clear in your question what you are trying to do or what result are you expecting. It seems, though, that you are trying to calculate a total with your variable z.

Check if the sum method produces the result that you need:

import numpy as np
nx = 3; ny = 5

ff = ff = np.array(np.arange(nx*ny*7)).reshape(nx,ny,7)

print ff.sum()          # 5460

print ff.sum(axis=0)    # array([[105, 108, 111, 114, 117, 120, 123],
                        #        [126, 129, 132, 135, 138, 141, 144],
                        #        [147, 150, 153, 156, 159, 162, 165],
                        #        [168, 171, 174, 177, 180, 183, 186],
                        #        [189, 192, 195, 198, 201, 204, 207]]) shape(5,7)

print ff.sum(axis=1)    # array([[ 70,  75,  80,  85,  90,  95, 100],
                        #        [245, 250, 255, 260, 265, 270, 275],
                        #        [420, 425, 430, 435, 440, 445, 450]]) shape (3,7)

print ff.sum(axis=2)    # array([[ 21,  70, 119, 168, 217],
                        #        [266, 315, 364, 413, 462],
                        #        [511, 560, 609, 658, 707]]) shape (3,5)