Does any open, simply extendible web crawler exists?

StackOverflow https://stackoverflow.com/questions/2085213

  •  21-09-2019
  •  | 
  •  

Question

I search for a web crawler solution which can is mature enough and can be simply extended. I am interested in the following features... or possibility to extend the crawler to meet them:

  • partly just to read the feeds of several sites
  • to scrap the content of these sites
  • if the site has an archive I would like to crawl and index it as well
  • the crawler should be capable to explore part of the Web for me and it should be able to decide which sites matches the given criteria
  • should be able to notify me, if things possibly matching my interest were found
  • the crawler should not kill the servers by attacking it by too many requests, it should be smart doing crawling
  • the crawler should be robust against freak sites and servers

Those things above can be done one by one without any big effort, but I am interested in any solution which provide a customisable, extendible crawler. I heard of Apache Nutch, but very unsure about the project so far. Do you have experiences with it? Can you recommend alternatives?

Was it helpful?

Solution

A quick search on GitHub threw up Anemone, a web spider framework which seems to fit your requirements - particularly extensiblility. Written in Ruby.
Hope it goes well!

OTHER TIPS

I've used Nutch extensively, when I was building the open source project index for my Krugle startup. It's hard to customize, being a fairly monolithic design. There is a plug-in architecture, but the interaction between plug-ins and the system is tricky and fragile.

As a result of that experience, and needing something with more flexibility, I started the Bixo project - a web mining toolkit. http://openbixo.org.

Whether it's right for you depends on the weighting of factors such as:

  1. How much flexibility you need (+)
  2. How mature it should be (-)
  3. Whether you need the ability to scale (+)
  4. If you're comfortable with Java/Hadoop (+)

I heartily recommend heritrix. It is VERY flexible and I'd argue is the most battle tested freely available open source crawler, as it's the one the Internet Archive uses.

You should be able to find something that fits your needs here.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top