Crawling sub-domain with Anemone

Question

I am using Anemone. How do I crawl sub-domain too? for e.g if I have website www.abc.com my crawler should also crawl support.abc.com or blah.abc.com. I am using Ruby 1.8.7 and Rails 3.

Answer 1

Here is a commit on Github that solves your problem.

https://github.com/runa/anemone/commit/91559bde052956cfc40ae62678ec2a61574cf928

Change your anemone gem files as per the link.

Answer 2

According to the Anemone docs you can pass multiple sites into the crawl command:

Anemone.crawl("http://www.abc.com/", "http://support.abc.com/", "http://blah.abc.com/")

Of course, your next problem will probably be ABC banning you for crawling their site, but that's a different question.