This is happening because the default interpolation of imshow
is bilinear. Setting it to 'none'
will probably make colors appear as you expect.
So add:
plt.imshow(mat, cmap=cm, interpolation='none')
Then it will look like:
If you increase the size of the matrix you will notice that the edge between red and blue is discrete as expected while the rest is interpolated: