Restricting URLs to seed URL domain only crawler4j

Question

I want crawler4j to visit pages in such a manner that they belong to domain in seed only. There multiple domains in seed. How can I do it?

Suppose I am adding seed URLs:

• www.google.com
• www.yahoo.com
• www.wikipedia.com

Now I am starting the crawling but I want my crawler to visit pages (just like shouldVisit()) only in above three domains. Obviously there external links, but I want my crawler to restrict to these domains only. Sub-domain, sub-folders are okay, but not outside these domains.

Answer 1

If you are trying to restrict the crawler to only urls with the same domains as the seed urls, then:

1. Extract the domain names from the seed URLs.

2. Write your crawler class (that extends WebCrawler) with a shouldVisit method to filter out any URLs whose domains are not in the set.

3. Configure the controller, add the seeds and start it in the normal way ... as per the example here.

Answer 2

Posted on behalf of the OP:

Got the solution here : http://code.google.com/p/crawler4j/issues/detail?id=94#c1