Why does WebClient timeout when trying to crawl this shortened URL?
Question
I have a small webcrawler that sometimes has to crawl twitter and pull out URL's. I use a modified version of the Webclient class provided in the .net framework.
Normally this works fine, even with shortened URL's from sites.
However, when following a shortened URL, the webclient times out.
You think they're filtering certain clients?
How I can fix this or why its happening?
Solution
Are you sure you can hit that URL from your network, without going through a proxy?
Does your webclient control follow redirects? You could test this by creating a TinyURL and see if your webclient can browse to it.
If you are going through a proxy in your browser, you'll need to set it up in the WebClient control.
It should be easy to test if they are filtering clients - set the UserAgent on the Request object to match that of FireFox for example.