It is bad enough that they allow covariance, because then every time you write to an array, i.e. set one of its elements to a new reference, a type check will have to be performed.
If they allowed contravariance as well, every read of the array would need a type check as well. That would be even worse.
In the old days of .NET arrays were often passed where the receiver was expected to only read from the array. The "crazy" covariance made a bit sense then. Contravariance was not as useful.