sock.recv() is taking too much time to execute in the python code

Question

Following is the code which listens on a port for HTTP requests and sends the request packet to the server running on port 80, gets the response and sends the data back to the client. Now, everything is executing fine but the following line of code :

data = req_soc.recv(1024)

is taking too much time to execute and I have observed that, it takes long time to execute when it is going to/has received the last packet. I have also tried the same code using select.select() but the results are the same. Since I want to handle the data (raw) that is coming from the client and the actual HTTP server, I have no other choice than using sockets.

import socket
import thread

def handle_client(client):
    data = client.recv(512)
    request = ''
    request += data
    print data

print '-'*20
spl = data.split("\r\n")

print spl[0]
print spl[1]

if len(request):
    req_soc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    req_soc.connect(('localhost', 80))
    req_soc.send(request)

    response = ''
    data = req_soc.recv(1024)
    while data:
        response += data
        print 1
        data = req_soc.recv(1024)
    req_soc.close()
    print response
    if len(response):
        client.send(response)
client.close()

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('localhost', 4422))
server.listen(5)

print("Server is running...\n")

MSGLEN = 1024

while 1:

    client, address = server.accept()
    thread.start_new_thread(handle_client, (client, ))

Answer 1

Clients can do multiple commands (eg: GET) within one connection. You cannot wait for the client to send all the commands because based on what you return it could request more (eg: images of a web page). You have to parse the parts (commands) of request, find the boundary, forward that request to the server and write back the answer to the client. All this in a way that doesn't block on reading the client.

I'm not sure what's the best way to do this in python, but if you spend 5 minutes of googling you'll find a perfect HTTP proxy library.