Non-blocking connect OpenSSL

Question

I created a regular C socket. Upon connect, it returns EWOULDBLOCK/WSAEWOULDBLOCK as expected because I did:

unsigned long int mode = 0;
ioctlsocket(ssl_info->sock, FIONBIO, &mode);
setsockopt(ssl_info->sock, SOL_SOCKET, SO_RCVTIMEO, (char*)&tv, sizeof(tv));
setsockopt(ssl_info->sock, SOL_SOCKET, SO_SNDTIMEO, (char*)&tv, sizeof(tv));

to put the socket in non-blocking mode. After that I do:

ssl = SSL_new(ctx);
SSL_set_fd(ssl, sock);
return SSL_connect(ssl);

However, it returns -1.

I read online that it means I need to handle SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE.

so I did:

int res = -1;
while(res == -1)
{
    res = SSL_connect(ssl);
    switch (SSL_get_error(ssl, res))
    {
        case SSL_ERROR_WANT_CONNECT:
        MessageBox(NULL, "Connect Error", "", 0);
        break;

        case SSL_ERROR_WANT_READ:   //prints this every time..
        MessageBox(NULL, "Read Error", "", 0);
        break;

        case SSL_ERROR_WANT_WRITE:
        MessageBox(NULL, "Write Error", "", 0);
        break;
    }

    SelectSocket(ssl);
}

std::cout<<"Connected!\n";

Where SelectSocket is defined as:

bool SelectSocket(SSL* ssl)
{
    if (blockmode)
    {
        fd_set readfds;
        fd_set writefds;
        FD_ZERO(&readfds);
        FD_ZERO (&writefds);
        FD_SET(ssl_info->sock, &readfds);
        FD_SET(ssl_info->sock, &writefds);

        struct timeval tv = {0};
        tv.tv_sec = timeout / 1000;
        tv.tv_usec = timeout % 1000;
        return select(sock + 1, &readfds, &writefds, NULL, &tv) >= 0;
    }

    return select(sock + 1, NULL, NULL, NULL, NULL) != SOCKET_ERROR;
}

So how exactly can I get it to connect? I can't seem to be able to read or write anything when the socket is non-blocking :S.

Any ideas?

Answer 1

The (-1) returned by SSL_connect() indicates that the underlying BIO could not satisfy the needs of SSL_connect() to continue the handshake.

Generally, the calling process then must repeat the call after taking appropriate action to satisfy the needs of SSL_connect().

However, when using a non-blocking socket, nothing is to be done; but select() can be used to check for the required condition.

(When using a buffering BIO, like a BIO pair, data must be written into or retrieved out of the BIO before being able to continue.)

Answer 2

Your code actually disables non-blocking I/O. As you are passing 0 as argument value for FIONBIO to ioctlsocket, which is documented as:

FIONBIO

The *argp parameter is a pointer to an unsigned long value. Set *argp to a nonzero value if the nonblocking mode should be enabled, or zero if the nonblocking mode should be disabled. [..]

https://msdn.microsoft.com/en-us/library/windows/desktop/ms738573%28v=vs.85%29.aspx