Acceptable rate of http requests

Question

I want to add a feature to my applications which parses an HTML page and jumps out to each <a href="http:\\foo" > in turn.

Im gathering title,description and keywords off each page load and loading nothing else and dont intend to allow the jump to be more than one in depth.

This in itself is a trivial application of NSURLDownload and NSXMLDocument but what I want to know...

Is there an acceptable rate of requests that wont make my app look like a DOS event as if I just flow out the child requests at the rate I parse them from the parent page then I suspect this will very rapidly get detected as such.

Answer 1

You should conform to the robots exclusion protocol. RFC 2616 specifies a limit of 2 concurrent requests per hostname, however modern browsers go past this and this part of the specification is likely to be revised in accordance. It's likely that NSURLConnection will use a connection limit internally anyway, but you should check this.