libssh2: What to do with unsolicited data from the ssh server?

Question

I have a program that uses libssh2 to administrate some Linux boxes. It's fairly straightforward: It connects to the Linux boxes, downloads a config file, then keeps the libssh2 connection open (and idle) so that if the user presses buttons on the GUI it can send the appropriate shell commands to the Linux boxes as necessary.

It uses non-blocking I/O (via libssh2_session_set_blocking(session, 0)) so that the network I/O can be handled by a single thread without session A's activity being held off my session B blocking on a read or write, etc.

This all mostly works fine. I say mostly because there is a strange problem that occurs only when the program is connected to many (i.e. several dozen) boxes at once. What happens is that the sessions all connect as usual, and the config files are downloaded successfully, and the connections are all idle, as expected. But on a few of the sessions, a few milliseconds after the download completes (i.e. after I've libssh2_channel_read()'d all the config file bytes, and after libssh2_channel_close() has succeeded), some additional bytes of data (usually 104 or 140) appear on the session's TCP socket, ready to be read by me and passed to libssh2.

At this point I have a problem, because (as far as I know) the libssh2 session should be idle, and I don't know what to do with these unsolicited bytes of data. None of my options are really satisfactory:

• I could just ignore the bytes, but then select() keeps waking up because the bytes are there, which causes the thread to spin and my CPU usage to rise significantly for as long as the program is open, so that's not good.

• I could stop select()-ing on ready-for-read for the sockets with the extra data, but then I can't tell if the socket gets closed by the remote peer, and I want to be able to notify the user when that happens.

• I could just recv() the excess bytes and throw them away, but that apparently screws up the state of the libssh2 connection's state machine because the next time I try to use it (e.g. when the user wants me to send a ssh command to the machine) the command does not get sent.

• I could do a sleep(1) or close the connection and reconnect it, but those would be ugly work-arounds to a problem that I think should be handled properly.

So my questions are

• What are these strange unsolicited bytes?

• What function should I call to pass them to libssh2 to make it happy?

ps Here are some examples of what the mystery received bytes looked like, obtained by doing a recv() with the MSG_PEEK flag:

--- peek'd data from session #1 (104 bytes): --------------------------------------------
0000: ..#|)62.!.A..... [ab 1f 23 7c 29 36 32 e8 21 cf 41 91 88 de 06 a4]
0016: xc.??n.+aJLS..di [78 63 f2 3f 3f 6e 16 2b 61 4a 4c 53 ab aa 64 69]
0032: ...d.]f.P...4;.3 [c2 d6 9e 64 b4 5d 66 db 50 ba 90 82 34 3b cc 33]
0048: cO"..5..Fr...Yy. [63 4f 22 ba b1 35 9c 00 46 72 a6 9c bb 59 79 a1]
0064: ..L....._..1.>.K [d2 fe 4c 2e e4 81 eb fd 5f 8e f2 31 da 3e c0 4b]
0080: ...........(d... [1d af df 0d 0f d1 ef 1e 07 d1 9f 28 64 f4 07 d3]
0096: 0...sl..         [30 b6 ee f7 73 6c cd 85]

--- peek'd data from session #2 (104 bytes): ----------------------------------------------
0000: g.5"....Q....... [67 d9 35 22 85 90 ab eb 51 95 11 0c e6 ca 9f de]
0016: ...)... .6V&.lkF [ed 04 cb 29 e9 87 95 20 85 36 56 26 a9 6c 6b 46]
0032: ...m).V....JS... [ce b2 d7 6d 29 bb 56 fc 8e 89 a2 4a 53 a9 02 19]
0048: .w1........kY... [a1 77 31 a1 fb f8 b7 94 ee e0 d3 6b 59 ea cc ae]
0064: ...~..[..vZ..... [e3 e8 f2 7e 2e 89 5b c4 82 76 5a da ff b6 ae 91]
0080: ......7.0Z..6g.J [8a cb c8 fc eb e1 37 8e 30 5a e5 b8 36 67 c7 4a]
0096: '...qJS.         [27 da b8 8f 71 4a 53 ef]

--- peek'd data from session #3 (140 bytes): -------------------------------------------------
0000: ...u.d.E..>a.... [04 c3 b1 75 e5 64 d1 45 9e b0 3e 61 81 e9 9b b7]
0016: ..\.....n..D..e. [e4 a2 5c b5 9e da a9 b9 6e 96 b7 44 12 bd 65 d9]
0032: .|(Jp..,k.....r' [c0 7c 28 4a 70 15 90 2c 6b 01 02 1a e6 d1 72 27]
0048: ..%..R]E:...N.CU [0b 8c 25 eb cd 52 5d 45 3a c4 12 f2 4e 11 43 55]
0064: gix$...d:.m..Ps1 [67 69 78 24 a1 e9 85 64 3a d5 6d 91 1a 50 73 31]
0080: Z.]...8#...q..AW [5a ca 5d db 1d 1d 38 23 0e 05 99 71 98 d8 41 57]
0096: ..]....U.7...J.? [ee 86 5d d1 0c b8 ce 55 f6 37 b5 1c 0f 4a b2 3f]
0112: ...10.......h(.* [17 15 ee 31 30 ea ee e0 b9 07 d1 c9 68 28 83 2a]
0128: ....>..g9...     [be c2 e4 f9 3e e2 ea 67 39 0f b2 8c]

Answer 1

Closing an SSH channel properly involves more steps than just "closing it" on one side.

---

First, the documentation for libssh2_channel_close() says it sends "an SSH_MSG_CLOSE packet to the remote host which serves as instruction that no further data will be sent to it", which is OK, but also that "the remote host may still send data back until it sends its own close message in response".

This command should be therefore followed by libssh2_channel_wait_closed(), which handles the rest of the communication "until the remote host closes the named channel".

---

Additionally, as described e.g. in comment for this libssh2 commit, "sending SSH_MSG_CHANNEL_CLOSE without channel EOF is explicitly allowed in RFC 4254, but some non-conforming servers will hang or time out when the channel is closed before EOF".

Thus performing another combo of libssh2_channel_send_eof() and libssh2_channel_wait_eof() before the close operation itself might be necessary too for compatibility.

Since libssh2 version 1.2.5 (April 13 2010), because of that commit mentioned earlier, libssh2 should send the EOF itself automatically when closing if not already sent.

---

There are some examples in libssh2 itself (like scp_write.c) which do the closure explicitly and it looks like this:

libssh2_channel_send_eof(channel);
libssh2_channel_wait_eof(channel);
libssh2_channel_wait_closed(channel);
libssh2_channel_free(channel);

---

(Of course, all of the functions above simply need to be invoked multiple times as long as they return LIBSSH2_ERROR_EAGAIN for non-blocking sockets.)