.Net based web crawler sample

Question

I am using VSTS 2008 + C# + .Net 3.5. I want to find a tool (open source) which crawls all web pages for a web site, and for any other domain pages which is linked by this web site, I want to skip to crawl them (I only need page for this specific domain only). For crawled web page, I want to store them into local file directory.

Any samples or ready to use open source tool?

Answer 1

Arachnode.net might be what you are looking for.

Answer 2

I am the author of AN.

AN indexes all languages by default. Nothing to configure.

• Mike