I did it by modifying the Control of the StreamSocket that way :
_streamSocket.Control.IgnorableServerCertificateErrors.Add(ChainValidationResult.Untrusted);
_streamSocket.Control.IgnorableServerCertificateErrors.Add(ChainValidationResult.InvalidName);
ChainValidationResult contains multiple options, choose the one best suited for you.