OpenSSL ssl_accept() error 5

Question

I have scoured Stack Overflow and the internet, but I have been unable to locate an answer to why ssl_accept() keeps returning:

[DEBUG] SSL_accept() : Failed with return 0
[DEBUG]     SSL_get_error() returned : 5
[DEBUG]     Error string : error:00000005:lib(0):func(0):DH lib
[DEBUG]     WSAGetLastError() returned : 0
[DEBUG]     GetLastError() returned : 0
[DEBUG]     ERR_get_error() returned : 0

Edit: Out of interest ssl_accept() returns 0, defined as (accordingly to the scant and unhelpful OpenSSL documentation): "The TLS/SSL handshake was not successful but was shut down controlled and by the specifications of the TLS/SSL protocol. Call SSL_get_error() with the return value ret to find out the reason."

Below is the snippet of the server side, could I be barking up the wrong tree and this issue be caused by client code?

    client = accept( server, (sockaddr*) &clientsockaddrin, &len );

    SSL* ssl = SSL_new( ctx );

    SSL_set_fd( ssl, client );

    std::cout << "+--------------------------------------------------+"
              << std::endl;

    int r = SSL_accept( ssl );

    if ( r != 1 ) 
    {
        int err_SSL_get_error = SSL_get_error( ssl, r);
        int err_GetLastError = GetLastError();
        int err_WSAGetLastError = WSAGetLastError();
        int err_ERR_get_error = ERR_get_error();

        std::cout << "[DEBUG] SSL_accept() : Failed with return " 
                  << r << std::endl;
        std::cout << "[DEBUG]     SSL_get_error() returned : "
                  << err_SSL_get_error << std::endl;
        std::cout << "[DEBUG]     Error string : "
                  << ERR_error_string( err_SSL_get_error, NULL ) 
                  << std::endl;
        std::cout << "[DEBUG]     WSAGetLastError() returned : "
                  << err_WSAGetLastError << std::endl;
        std::cout << "[DEBUG]     GetLastError() returned : "
                  << err_GetLastError << std::endl;
        std::cout << "[DEBUG]     ERR_get_error() returned : "
                  << err_ERR_get_error << std::endl;
        std::cout << "+--------------------------------------------------+"
                  << std::endl;
        break;
    }

Appreciate your assistance as this is driving me mad :(

Answer 1

[DEBUG] Error string : error:00000005:lib(0):func(0):DH lib

The error happened during the Diffie-Hellman Key Exchange, e.g. where the peers tried to generate the keys for the connection. There might be several reasons for this, like invalid DH parameters given on the server side. With the your current code it is hard to see where the error actually is, but I guess is somewhere in setting up your ctx, so maybe should show the relevant parts of the code.

Answer 2

This is not a Diffie-Hellman library issue.
The reason you are getting the

error:00000005:lib(0):func(0):DH lib

is that you passed in the SSL_get_error() error code to ERR_error_string() which you should not do.

ERR_error_string() is only used on error codes from ERR_get_error().
See the help page for SSL_get_error() to know what the error means.