Why does WebClient timeout when trying to crawl this shortened URL?

Question

I have a small webcrawler that sometimes has to crawl twitter and pull out URL's. I use a modified version of the Webclient class provided in the .net framework.

Normally this works fine, even with shortened URL's from sites.

However, when following a shortened URL, the webclient times out.

You think they're filtering certain clients?

How I can fix this or why its happening?

Answer 1

Are you sure you can hit that URL from your network, without going through a proxy?

Does your webclient control follow redirects? You could test this by creating a TinyURL and see if your webclient can browse to it.

If you are going through a proxy in your browser, you'll need to set it up in the WebClient control.

It should be easy to test if they are filtering clients - set the UserAgent on the Request object to match that of FireFox for example.