Python, Catch timeout during stream request

Question

I'm reading XML events with the requests library as stated in the code below. How do I raise a connection-lost error once the request is started? The Server is emulating a HTTP push / long polling -> http://en.wikipedia.org/wiki/Push_technology#Long_polling and will not end by default. If there is no new message after 10minutes, the while loop should be exited.

import requests
from time import time


if __name__ == '__main__':
    #: Set a default content-length
    content_length = 512
    try:
        requests_stream = requests.get('http://agent.mtconnect.org:80/sample?interval=0', stream=True, timeout=2)
        while True:
            start_time = time()
            #: Read three lines to determine the content-length         
            for line in requests_stream.iter_lines(3, decode_unicode=None):
                if line.startswith('Content-length'):
                    content_length = int(''.join(x for x in line if x.isdigit()))
                    #: pause the generator
                    break

            #: Continue the generator and read the exact amount of the body.        
            for xml in requests_stream.iter_content(content_length):
                print "Received XML document with content length of %s in %s seconds" % (len(xml), time() - start_time)
                break

    except requests.exceptions.RequestException as e:
        print('error: ', e)

The server push could be tested with curl via command line:

curl http://agent.mtconnect.org:80/sample\?interval\=0

Answer 1

This might not be the best method, but you can use multiprocessing to run the requests in a separate process. Something like this should work:

import multiprocessing
import requests
import time

class RequestClient(multiprocessing.Process):
    def run(self):
        # Write all your code to process the requests here
        content_length = 512
        try:
            requests_stream = requests.get('http://agent.mtconnect.org:80/sample?interval=0', stream=True, timeout=2)

            start_time = time.time()
            for line in requests_stream.iter_lines(3, decode_unicode=None):
                if line.startswith('Content-length'):
                    content_length = int(''.join(x for x in line if x.isdigit()))
                    break

            for xml in requests_stream.iter_content(content_length):
                print "Received XML document with content length of %s in %s seconds" % (len(xml), time.time() - start_time) 
                break
        except requests.exceptions.RequestException as e:
            print('error: ', e)


While True:
    childProcess = RequestClient()
    childProcess.start()

    # Wait for 10mins
    start_time = time.time()
    while time.time() - start_time <= 600:
        # Check if the process is still active
        if not childProcess.is_alive():
            # Request completed
            break
        time.sleep(5)    # Give the system some breathing time

    # Check if the process is still active after 10mins.
    if childProcess.is_alive():
        # Shutdown the process
        childProcess.terminate()
        raise RuntimeError("Connection Timed-out")

Not the perfect code for your problem, but you get the idea.