python sending incomplete data over socket

Question

I have this socket server script,

import SocketServer
import shelve
import zlib

    class MyTCPHandler(SocketServer.BaseRequestHandler):
        def handle(self):
            self.words = shelve.open('/home/tipu/Dropbox/dev/workspace/search/words.db', 'r');
            self.tweets = shelve.open('/home/tipu/Dropbox/dev/workspace/search/tweets.db', 'r');

            param = self.request.recv(1024).strip()
            try:
                result = str(self.words[param])
            except KeyError:
                result = "set()"
            self.request.send(str(result))

    if __name__ == "__main__":
        HOST, PORT = "localhost", 50007
        SocketServer.TCPServer.allow_reuse_address  = True
        server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
        server.serve_forever()

And this receiver,

from django.http import HttpResponse
from django.template import Context, loader
import shelve
import zlib
import socket


def index(req, param = ''):
    HOST = 'localhost'    
    PORT = 50007              
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect((HOST, PORT))
    s.send(param)
    data = zlib.decompress(s.recv(131072))
    s.close()
    print 'Received', repr(data)
    t = loader.get_template('index.html') 
    c = Context({ 'foo' : data })
    return HttpResponse(t.render(c))

I am sending strings to the receiver that are in the hundreds of kilobytes. I end up only receiving a portion of it. Is there a way that I can fix that so that the whole string is sent?

Edit: The string I am trying to send is actually 418437 characters. But here's the idea.. I'm trying to send the string representation of a set, using str(set), so I can reassemble the set using eval. Is there a way can I either get the socket to send the complete data or compress it enough so that the socket can send it at once? I tried using zlib, but the compressed data doesn't send completely either because I repeatedly get header errors which, from what I was able to find, is because the compressed string was incomplete.

Answer 1

socket.recv(N) does not guarantee that exactly N bytes will be received: rather, N is just the maximum number of bytes that can be received at one gulp (and for efficiency it should ideally be a small multiple of 4096, as the docs suggest).

You'll need to "keep receiving" (looping and accumulating) until you have all the bytes you desire (and it looks like your protocol does not communicate that crucial value: you will have to make it explicit, either with a prefix length or a terminator at the end of transmission, there is no way it can be implicitly extracted from the stream socket).

Similarly for sending, although for that case you can use socket.sendall which does the necessary looping on your behalf.

Answer 2

Use pickle module to serialize the object onto the socket and load it on the other side. Something like:

    pkl_dict= pickle.dumps(THE_SET)
    mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    mysocket.connect((HOST, PORT))

    mysocket.send(pkl_dict)

    mysocket.close()