Yes, this is considered secure because the fingerprint of the server's certificate is transferred to the client via a trusted mechanism (being physically next to the server to receive the fingerprint via a difficult to tamper with transport mechanism). If a MITM attack were to be attempted after the initial setup, even though server name would be the same, different keys would have been generated so the fingerprint would be different and the client would detect this and can appropriately reject the communication.
It could also be argued that this method is more secure because the client does not need to trust a 3rd party to verify the authenticity of the certificate. CAs do occasionally issue fraudulent certificates.