Is this correct? Does the fact that a single number can only be in 21 spaces out of the 25 at any single moment not affect it?
No, it's not correct. The difference to a Go board is that you only have 5 numbers and thus always 20 of the 25 places are empty, and each non-empty state can appear only once.
So there are 25 `choose` 5
possibilities for the five places where the numbers are, and the five numbers can be arranged in 5!
ways on these five places.
Thus you have a total state space of
25!/20! = 21*22*23*24*25 = 6375600
states.