SMTPS: OpenSSL - SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol s23_clnt.c:787:

Question

I'm using OpenSSL in order to encrypt some emails, that a piece of hardware sends. But, whenever I try to call SSL_connect(), I get : SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

After sending "EHLO" and "STARTTLS" I call the following function:

SSL_CTX *ctx = NULL;
SSL *ssl = NULL;

    void CreateTLSSession(int sockfd)
    {
        printf("///////////////creating TLS Session/////////////////////\n");
        SSL_library_init();
        SSL_load_error_strings();
        OpenSSL_add_all_algorithms();
        ctx = SSL_CTX_new(SSLv23_client_method());
        if (ctx == NULL)
        {
            printf("failed to initialize context\n");
            return;
        }
        SSL_CTX_set_options(ctx, SSL_OP_NO_SSLv2);
        ssl = SSL_new(ctx);
        if (!SSL_set_fd(ssl, sockfd))
        {
            printf("failed to bind to socket fd\n");
            return;
        }
        if (SSL_connect(ssl) < 1)
        {
            ERR_print_errors_fp(stdout);
            fflush(stdout);
            printf("SSL_connect failed\n");
            return;
        }
    }

I've tried connecting to :

• smtp.live.com : 587 --> SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol s23_clnt.c:787:
• smtp.live.com : 25 --> SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol s23_clnt.c:787:
• smtp.gmail.com : 587 --> SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol s23_clnt.c:787:
• smtp.gmail.com : 465 --> no response from server at all!
• smtp.gmail.com : 25 --> SSL routines:SSL23_GET_SERVER_HELLO:unknown
  protocol s23_clnt.c:787:

I've tried different ports, since some similar questions on this SO suggested, that such error is usually related to using the wrong port for SSL.

Am I missing something here?

UPDATE:

All other methods (i.e. TLSv1_1_method(), SSLv3_method() ...) lead to SSL3_GET_RECORD:wrong version number

UPDATE:

I was able to observe the following on wireshark:

"EHLO"

"at your service"

"STARTTLS"

"Ready to starttls"

-->now I call the above function

unreadable request (encrypted)

unreadable reply (encrypted)

--> ERROR

Answer 1

The underlying socked was non-blocking. The problem was solved, by using select and waiting till the TLS handshake completes.

Answer 2

SMTP servers on ports 587 and 25 are usually plain text and will switch to TLS only after the initial SMTP dialog and a STARTTLS command from the client. And trying SSL_connect on the plain text socket will fail.

Answer 3

Another way to solve this problem may be to run your C program under Scott Gifford's sslclient (see http://www.superscript.com/ucspi-ssl/sslclient.html). sslclient will spawn your program and open an tcp connection to the server, and pipe your program's stdout to the server, and pipe output from the server to your program's stdin. He has a patched version for TLS that will start the connection off in plain text, then once the two sides have agreed on STARTTLS, your program can signal to sslcient to enable SSL encryption on the connection by writing a command to a file descriptor for this purpose. The nice thing about doing it this way is that you can let sslclient to all the heavy lifting as far as setting up the sockets and ssl, etc., and you can focus on the core function of your program.