It really depends on the site and services it provides and configuration it has. Things that may help to look for (not a complete list):
- API exposed that let you access data directly. E.g. XML or JSON type response.
- Compression - your client has to request via appropriate HTTP headers, e.g.
Accept-Encoding: gzip, deflate
, and needless to say know how to process response accordingly. SO thread on doing this in C#. - Requesting mobile version of site if site supports such a thing. How site exposes such version really depends on the site. Some prefix their URLs with
m.
, some respond toUser-Agent
string, some use other strategies... - Use HTTP
Range
header. Also depends if site supports it. MSDN link for .NET API.