python-requests hook similar to pycurl.WRITEFUNCTION?

Question

I've been utilizing PycURL for my project https://pypi.python.org/pypi/tidehunter which makes use of the pycurl.WRITEFUNCTION hook to consume incoming HTTP stream in a controllable way. Use cases can be found on the project page, or the technique like this:

def _on_data_receive(self, data):
    self.buffer += data

    if data.endswith('\r\n'):  # or any valid chunk delimiter
        # Do something about self.buffer
        print self.buffer

    if True:  # some valid condition to disconnect
        return -1  # the way to stop the stream, would raise pycurl.error

I'm considering switching out of PycURL as the solution I'm using to terminate stream on-demand is rather hacky (also happy to gain better solution for that use case with PycURL). Also requests is a much more pleasant library to use.

So is there anything in requests that I can leverage to achieve the same purpose? Or maybe it's also something like a incoming stream handler that I need to explore.

Thanks in advance.

Answer 1

EDIT 2016-03-22: Apologize for the selfishness.

Current Answer

The following snippet is what I applied to fully replace pycurl using python-requests. In short, simply supply with stream=True argument.

import requests

def tide_on(url):
    r = requests.get(url, stream=True, **kwargs)

    for line in r.iter_lines():
        # wish to stop after consuming just one line
        print(line)
        break

    r.close()

---

Previous "Answer"

The current version of requests at the time of speaking really made it easy to implement what I asked. Huge thanks to the team behind requests!

I fully replaced PycURL with requests in tidehunter since version 1.0 and it's working great!

Thanks again to all the comments above :)