ColdFusion - How to limit data retrieved via CFHTTP?

Question

I am using cfhttp to get documents and parse the content like so:

<cfhttp
        method="GET"
        url="#url.strURL#"
        resolveurl="true"
        useragent="#CGI.http_user_agent#"
        result="objGet"
        timeout="60"
        charset="utf-8"
    />

However, some of the documents are rather large. I do not need to get the entire thing - which could take a long time.

Is there any way in which I can stop running cfhttp after X number of bytes loaded? Or set a limit on how much to get for example.

Appreciate the help.

Answer 1

If the target server supports it, you can use the Range http header:

<cfhttp
        method="GET"
        url="#url.strURL#"
        resolveurl="true"
        useragent="#CGI.http_user_agent#"
        result="objGet"
        timeout="60"
        charset="utf-8"/>
   <cfhttpparam type="header" name="Range" value="bytes=0-499" />
 </cfhttp>

Apache and IIS supports this, so for static content you'll probably be in luck. Dynamic content would be trickier...

...I've just tried this with a CFM being served through Apache, and it looks like that didn't work. I suspect that if Apache/IIS is serving a static file, then it can safely send back a range of bytes from the document you're requesting because it can read the file. If the request is for something being generated by CF/ASP/JSP/whatever, then it'd be up to the Appliation to honour the Range: header in the request. In my case, I still got the whole document because my Application doesn't look at the Range header.

I should also mention that it's possible to do what you're after in Java, as you've got finer-level control of what's going on, but you'd have to write something yourself or use one of the alternative HTTP Client Libraries out there. It depends how important the feature is to you I suppose. If the documents are seriously large then you may need to use the file attribute of cfhttp to avoid getting the whole response into memory, which could cause crashes.