What is causing OpenSSL to segfault?

Question

(gdb) bt
#0  0x040010c2 in ?? () from /lib/ld-linux.so.2
#1  0x06a14a0b in write () at ../sysdeps/unix/syscall-template.S:82
#2  0x04154ae9 in ?? () from /lib/i386-linux-gnu/libcrypto.so.1.0.0
#3  0x041518e4 in BIO_write () from /lib/i386-linux-gnu/libcrypto.so.1.0.0
#4  0x040781f1 in ?? () from /lib/i386-linux-gnu/libssl.so.1.0.0
#5  0x040785ff in ?? () from /lib/i386-linux-gnu/libssl.so.1.0.0
#6  0x04078855 in ?? () from /lib/i386-linux-gnu/libssl.so.1.0.0
#7  0x04075e28 in ?? () from /lib/i386-linux-gnu/libssl.so.1.0.0
#8  0x0408d709 in SSL_write () from /lib/i386-linux-gnu/libssl.so.1.0.0
#9  0x0409c451 in ?? () from /lib/i386-linux-gnu/libssl.so.1.0.0
#10 0x041518e4 in BIO_write () from /lib/i386-linux-gnu/libcrypto.so.1.0.0
#11 0x0814b10f in SSL_Connection_send (connection=0x9ffbbd0) 
...

(gdb) print *connection->bio
$1 = {method = 0x40ac800, callback = 0, cb_arg = 0x0, init = 1, shutdown = 1, flags = 0, retry_reason = 0, num = 0, ptr = 0xa27e768, next_bio = 0x7b84ad0, prev_bio = 0x0, references = 1, num_read = 904, 
  num_write = 2870, ex_data = {sk = 0x0, dummy = 774321733}}

(gdb) print *connection->ssl
$2 = {version = 769, type = 4096, method = 0x40aacc0, rbio = 0x7b84ad0, wbio = 0x7b84ad0, bbio = 0x0, rwstate = 2, in_handshake = 0, handshake_func = 0x40738f0, server = 0, new_session = 0, quiet_shutdown = 0, 
  shutdown = 0, state = 3, rstate = 240, init_buf = 0x0, init_msg = 0xf5838e4, init_num = 0, init_off = 0, 
  packet = 0xf848ebb "\027\003\001\001\330HTTP/1.1 200 OK\r\nAccess-Control-Allow-Origin: *\r\nCache-Control: private, no-cache, no-store, must-revalidate\r\nContent-Type: text/javascript; charset=UTF-8\r\nETag: \"cacca674ed49d64124f812372ad59561"..., packet_length = 0, s2 = 0x0, s3 = 0xf3ea410, d1 = 0x0, read_ahead = 0, msg_callback = 0, msg_callback_arg = 0x0, hit = 0, param = 0x99f82b8, cipher_list = 0x0, 
  cipher_list_by_id = 0x0, mac_flags = 0, enc_read_ctx = 0xace5438, read_hash = 0xabce1c8, expand = 0x0, enc_write_ctx = 0x9794468, write_hash = 0xc057018, compress = 0x0, cert = 0xe1f70d8, sid_ctx_length = 0, 
  sid_ctx = '\000' , session = 0x7d54760, generate_session_id = 0, verify_mode = 0, verify_callback = 0, info_callback = 0, error = 0, error_code = 0, psk_client_callback = 0, 
  psk_server_callback = 0, ctx = 0xae8ce30, debug = 0, verify_result = 20, ex_data = {sk = 0x0, dummy = 0}, client_CA = 0x0, references = 1, options = 4, mode = 4, max_cert_list = 102400, first_packet = 0, 
  client_version = 769, max_send_fragment = 16384, tlsext_debug_cb = 0, tlsext_debug_arg = 0x0, tlsext_hostname = 0x0, servername_done = 0, tlsext_status_type = -1, tlsext_status_expected = 0, 
  tlsext_ocsp_ids = 0x0, tlsext_ocsp_exts = 0x0, tlsext_ocsp_resp = 0x0, tlsext_ocsp_resplen = -1, tlsext_ticket_expected = 1, tlsext_ecpointformatlist_length = 3, tlsext_ecpointformatlist = 0xac40bc8 "", 
  tlsext_ellipticcurvelist_length = 50, tlsext_ellipticcurvelist = 0x7b20878 "", tlsext_opaque_prf_input = 0x0, tlsext_opaque_prf_input_len = 0, tlsext_session_ticket = 0x0, tls_session_ticket_ext_cb = 0, 
  tls_session_ticket_ext_cb_arg = 0x0, tls_session_secret_cb = 0, tls_session_secret_cb_arg = 0x0, initial_ctx = 0xae8ce30, next_proto_negotiated = 0x982fd50 "Groups.History", 
  next_proto_negotiated_len = 111 'o', srtp_profiles = 0x7373656d, srtp_profile = 0x2e656761, tlsext_heartbeat = 1953720648, tlsext_hb_pending = 0, tlsext_hb_seq = 424, renegotiate = 1232, srp_ctx = {
    SRP_cb_arg = 0x0, TLS_ext_srp_username_callback = 0x6f697463, SRP_verify_param_callback = 0x64695f6e, SRP_give_srp_client_pwd_callback = 0x4c4f202c, login = 0x72742e44 "", N = 0x498, g = 0x0, s = 0xa, 
    B = 0xfc0301, A = 0xfc03, a = 0x0, b = 0x0, v = 0x0, info = 0x0, strength = 0, srp_Mask = 0}}

I am trying to figure out why this is erroring. The error happens randomly when I make a request to a server (only on linux too). Could it be that the remote side has closed the connection (connection->bio->shutdown == 1)? Or is it a memory error (next_proto_negotiated = 0x982fd50 "Groups.History"...that doesnt look like a protocol, although valgrind didnt pick up any memory error).

Answer 1

Are you doing any multi-threaded programming, accessing the socket on different threads at the same time? If so, make sure you're not writing to it on one thread and closing it on another. I've made that mistake before and seen crashes just like this. It took some annoying rearchitecting to get all the SSL calls on one thread.

Answer 2

If you are facing issues because of multi-threading, it might help to sleep for a second between two calls to SSL_shutdown(). The following article does exactly that even in a single threaded program.

http://pic.dhe.ibm.com/infocenter/tpfhelp/current/index.jsp?topic=%2Fcom.ibm.ztpf-ztpfdf.doc_put.cur%2Fgtps5%2Fs5sple2.html

Answer 3

I have seen openssl on Linux segfault if the server neglects to call SSL_CTX_set_session_id_context when setting up the TLS context.

A good indication that this is causing the crash is if a newer TLS client, such as Windows Server 2012, is trying to reconnect with TLS tickets. What you will see is every other connection attempt causing a segfault, as after a good connection, the client comes back for a second connection using a TLS ticket.

Even if you are not using TLS tickets or session caching, see if adding a call to SSL_CTX_set_session_id_context solves the issue.