how to get the content of a particular url using nutch database

Question

I am new with nutch concept. I have configured everything properly best of my knowledge. Am able to crawl the links, and i can get crawled urls also.

My problem is that, want to fetch content of webpages separate for every link, and am not able to find the solution for it.

Can anyone please help me??

Thank you.

Answer 1

I have seperate the files through a logic..as i am able to get the content for all urls in a single file with a particular pattern repeating for every record or url..i have seperate the content on row field.

Answer 2

Use the nokogiri gem http://rubygems.org/gems/nokogiri for parsing the content of webpages and select the link using nokogiri selector