A) initially, a server-to-server smtp connection is always in plain text on port 25. if both source and target server support the TLS extension then usually the plain connection gets converted into a encrypted connection with the STARTTLS command
B) To check if a mail was transmitted over an encrypted connection, read the "Received"
-Headers in the resulting message after it was transmitted.
They look like this:
Received: from X.example.com (X.example.com [y.y.y.y])
by z.example.net (Postfix) with ESMTPS id ......
The important part is the ESMTPS bit. The last S
means "SECURED".
If it just says "ESMTP" or "SMTP" instead of "ESMTPS" the transmission was not encrypted.
C) if the target server does not support TLS there is nothing you can do except some sort of end-to-end encryption like PGP (as suggested by Álvaro G. Vicario). Some servers (like postfix) provide configuration options to prevent messages from going out at all if the target can not do TLS.
you can test manually with telnet if a server supports STARTTLS:
telnet gmail-smtp-in.l.google.com 25
Trying 173.194.70.27...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP 4si1878861eee.197 - gsmtp
EHLO mail.example.com <--- you have to type that
250-mx.google.com at your service
250-SIZE 35882577
250-8BITMIME
250-STARTTLS <----- GMAIL supports TLS
250 ENHANCEDSTATUSCODES