Nokogiri gem very slow on simple request?

Question

I am trying to use Nokogiri to scrape a little bit of information from Amazon.

For some reason it takes about 50 seconds for this process to complete. Am I doing something wrong that is making the code inefficient, or is that just how long it would take?

url = "http://www.amazon.com/dp/" + item["ASIN"]
page = Nokogiri::HTML(open(url))
@numreviews = page.css('#averageCustomerReviews > a').text

I have this in a loop that does this to twenty different URLs. Is there anyway to speed this up, or is it just that the pages it has to load to get the information will take that long?

Answer 1

This can be because the response of this URL is too big. You can use the Amazon API which will give you selected information in XML, which will be easy to parse as well.