Why don't you try to use [hostfingerprints] in your ~/.hgrc?
For example:
[hostfingerprints]
example.com = 38:76:52:7c:87:26:9a:8f:4a:f8:d3:de:08:45:3b:ea:d6:4b:ee:cc
To obtain the self-signed certificate, follow the steps suggested in https://www.mercurial-scm.org/wiki/CACertificates
The root CA certificate for a server can for example be retrieved with Firefox. Browse to https://example.com/repo and verify that this is the repository you trust, click the lock symbol in the lower right corner, View Certificate, Details, select the certificate at the top of the Certificate Hierarchy, Export, "X.509 Certificate (PEM)" and save somewhere for example as server.pem. With other browsers on Windows you have to view the certificate at the top of the Certification Path and "Copy to File" as "Base-64 encoded X.509 (.CER)". Several such files can be concatenated into one cacerts file.