struct.pack() Python unknown fields length to send through a socket

Question

I am developing a TCP client with Python. My problem comes when I want to send the structure to the server, I use the method struct.pack() to send and struct.unpack() to receive. I have my own protocol over TCP, with some headers which I already know the length of them, but of course, I don't know the length of the body, How could I pack all the fields without knowing the length of the body? Here is an example:

Request to the server:

pack('!BBBBHHII', P2P_h.version, P2P_h.ttl, P2P_h.msgType, P2P_h.reserved, P2P_h.sendPort, P2P_h.payloadLength, P2P_h.ipSrc, P2P_h.messageId)

Response from the server:

queryHit = unpack ("!BBBBHHII", queryResponse)

In those examples there are just the headers, I mean, the fields that I already know their length, how could I add the body field?

Answer 1

Python socket's send and `recv operate on strings. By using pack, you're already transforming other kinds of types (integers, booleans...) into strings. Assuming your body is just raw binary data - a string, that is - you just need to concatenate the string you already have with it. Also, since the body is of variable length, you probably want to also send the body size in the message, so that you know what size to read in recv:

data=pack('!BBBBHHII', P2P_h.version, P2P_h.ttl, P2P_h.msgType, P2P_h.reserved, P2P_h.sendPort, P2P_h.payloadLength, P2P_h.ipSrc, P2P_h.messageId) #your current header
data+= pack("!I",len(body)) #assumes len(body) fits into a integer
data+= body #assumes body is a string
socket.send(data)

To receive, you do the reverse:

data= socket.recv( struct.calcsize('!BBBBHHII') )
P2P_h.version, P2P_h.ttl, P2P_h.msgType, P2P_h.reserved, P2P_h.sendPort, P2P_h.payloadLength, P2P_h.ipSrc, P2P_h.messageId= unpack('!BBBBHHII', data)
data= socket.recv( struct.calcsize("!I") )
body_size= unpack("!I", data)
body= socket.recv( body_size )

I separated the body size from the other fields for clarity, but you can combine them.