SSLEngine
uses the chooseEngineServerAlias
method to pick the certificate to use (in server mode) - not the chooseServerAlias
method.
The default chooseEngineServerAlias
implementation actually returns null
, which is what causes the "no cipher suites in common" message - you need a certificate to know which cipher suites can be used (e.g. ECDSA can only be used for authentication if the certificate has an ECC public key, etc.) There are actually some cipher suites which can be used without a certificate, however, these are typically disabled as they are vulnerable to MITM attacks.
Therefore, you should also override chooseEngineServerAlias
, and implement your logic to select the certificate based on the IP address there. As Netty only uses SSLEngine
, what chooseServerAlias
does doesn't matter - it'll never be called.
Java 8 also has support for server-side SNI, which allows you to use several certificates across many hostnames with a single IP address. Most web browsers support SNI - the notable exceptions are IE running on Windows XP and some old versions of Android, however, usage of these is declining. I have created a small example application demonstrating how to use SNI in Netty on GitHub. The core part of how it works is by overriding chooseEngineServerAlias - which should give you enough hints, even if you want to use the one certificate per IP address technique instead of SNI.
(I posted a similar answer to this on the Netty mailing list, where you also asked this question - however, my post seems to have not yet been approved, so I thought I'd answer here too so you can get an answer sooner.)