Which seems to mean that the replica set is not failing over.
It's not the replica set that should fail over but the driver.
And what tells you it doesn't? The warning just informs you that one of the nodes is in DOWN
state. You haven't done any operation yet, hence nothing to fail over from.
Did you try to read or write beyond connecting to the set? I'm pretty much convinced you can.
I guess I must be doing something wrong
No you don't, except for understanding the meaning of the warning