It simplifies the parser. It can tell from the first character in a token whether it's an identifier or a number.
Also, the syntax for floating point can look like this:
123e45
This means 123x1045. If identifiers could start with a number, this could be confused with a variable.
There are some languages that don't have this prohibition, Common Lisp for instance. It's rule is essentially that the token is a symbol unless it can be parsed as a number. Since it also allows the input radix to be customized, it has the property that whether a token is a number or symbol depends on the setting of a variable (it also has escaping mechanisms that allow you to force it one way or the other).