I came here to suggest using MailKit instead of writing your own library for this. MailKit is also specifically meant to work with Xamarin (since I work at Xamarin). MailSystem.NET is pretty badly broken (I've ranted about it elsewhere on StackOverflow), so I would definitely not recommend using that.
That said, you may need to look at using this version of the SslStream .ctor as opposed to the one you are using. The problem may be that the default .ctor isn't validating the SSL certificate because it isn't "trusted".