Using C# to efficiently pull data from a webpage with changing sourcecode?

Question

I have already put together code using the System.net.Webclient class to pull source code from a webpage, which I then use a string search on, to get specific information. This in itself works fine, but my issue is that the source code changes every few seconds, and I would like the data I have received to change accordingly. I understand that I could simply set up a loop to have this process repeat, but unfortunately my current code take a full 2.7 seconds to complete, and I would like to avoid this large lag time. In addition, I want to avoid spamming the webpage with requests if possible. I was thinking about a streamread that stays open, so that multiple requests wouldn't have to be sent, but I wasn't entirely sure how to go about this...

So to sum it up, is there a way that I can pull updating information from a website using the System.Net namespace in a manner that is both fast, and avoids spamming the website with requests?

Answer 1

I am afraid that HTTP protocol is not adapted to your real-time data refresh requirement. Other than polling with HTTP requests at regular intervals you cannot know whether the data changed on the server and get this fresh data.

For example the WebSocket technology is more adapted to those scenarios. Of course the data provider must implement it so that clients could subscribe to this live feed.

There's also another way to implement this feature over the HTTP protocol. It uses an iframe to implement long polling. Here's an example. The idea is that the server uses chunked transfer encoding and sends continuous streams of data to the socket. The client subscribes to this stream and is able to be notified of changes occurring on the server. Once again, it's a technology that must be implemented by the server side so that you, as a client, could take advantage of it.

If all that the server provides is data via HTML page you are doomed to do screen scraping by hammering this server with HTTP requests until your IP address gets black listed and denied access.