Python Recv() stalling

Question

I'm writing a very basic HTTP client:

import socket
from socket import *

Payload = """GET /test.html HTTP/1.1
Accept: */*
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
Accept-Encoding: gzip, deflate
Proxy-Connection: Keep-Alive
Host: example.com
Pragma: no-cache

"""

def SendAndReceive(Host, Payload):
    s = socket(AF_INET, SOCK_STREAM)
    s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
    s.connect(Host)    
    s.sendall(Payload)
    tdata=[]
    while True:
        data = s.recv(1024)
        if not data: 
           break
        tdata.append(data)
    print ''.join(tdata)
    return ''.join(tdata)

SendAndReceive(("www.example.com",80),Payload)

For some reasons, the recv() stalls for a while (~10 sec), then returns the data. I'm not sure what's wrong with my code, any help would be greatly appreciated.

Thanks!

Answer 1

You do an HTTP/1.1 request which implicitly means keep-alive, e.g. the server can keep the connection open after the request is done to get more requests on the same connection. In this case the server decides after 10 seconds that it will not wait for more requests, other servers might wait longer. You also don't check the servers response for content-length or chunked encoded but just assume, that the server will close once the request is done. This is wrong (keep-alive). Also, you have a proxy-connection header which does nothing, because this is not a proxy request (and proxy-connection header in itself is invalid, talking to a proxy requires a connection header).

The easiest thing would be to issue a HTTP/1.0 request and not specifying any connection header or setting it to 'close'. Then the server will close the connection after the response is done. Also, you don't have to deal in this case with chunked encoding.

Answer 2

As explained by Steffen, this is due to the keep-alive. For example, if you test it with google.com, you will wait for a very long time. You can modify your code and see the timing out as follows:

# Payload remains the same
import socket as socket_module
from socket import *
def SendAndReceive(Host, Payload):
    s = socket(AF_INET, SOCK_STREAM)
    s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
    # or you could do s.setblocking(0) for timeout of 0 seconds
    s.settimeout(1)
    s.connect(Host)
    s.sendall(Payload)
    tdata=[]
    while True:
        try:
            data = s.recv(1024)
        except socket_module.error:
            # TIMEOUT
            break
        if not data:
            break
        tdata.append(data)
    print ''.join(tdata)
    return ''.join(tdata)

In this way you wouldn't get an error.