How to retrieve top 10 Google search results from a keyword through an API in Ruby?

Question

I'm a little confused as to what is the best way to retrieve the top 10 Google results for a specific search/keyword. I just need the title and url (description is not essential).

I'm using Ruby and apparently there was a great way to do it with the googleajax gem. I've been able to get it to work but am concerned that it's a deprecated API and could be phased out any day. Also, the workaround to get more than 4 results at a time isn't very clean.

I think the Google Custom Search might be an option but the daily limit of 100 queries is restricting. I would prefer to not scrape Google as it's a violation of their terms.

What other options do I have to make this work? Any json/ruby/rails option would work for me. Thank you!

Answer 1

You're not very explicit in your question about the trade offs you're willing to make, But you might want to think about this more:

I think the Google Custom Search might be an option but the daily limit of 100 queries is restricting. I would prefer to not scrape Google as it's a violation of their terms.

I've used google custom search, and it is very easy but the limit is in place. If you are concerned about not violating Google's TOS, this is the only way to go. You need to decide if you're willing to violate the TOS, and if not you should just use the google custom search.

Answer 2

We ended having the same issue, and we built our own gem with our own backend. It's pretty simple to use:

query = GoogleSearchResults.new q: "coffee"
hash_results = query.get_hash

https://github.com/serpapi/google-search-results-ruby

Answer 3

I recommend use 'rest-client' gem.

RestClient.get 'google_api_url' 

Answer 4

it occurred to me first, just example:

require 'open-uri'
require 'nokogiri'
require 'restclient'

words = ["Foo", "Bar", "Baz"]
staff = [].tap do |acc|
  words.each do |word|
    link = "https://www.google.com/search?q=#{word}"
    page = Nokogiri::HTML(RestClient.get(link))
    page.css('a')[27].text
    .....................# <- and parse data what you need
  end
end

Answer 5

If your requirement is more than 100 queries on a single day, but not on a daily/regular basis and if Google results are not a must have, you can consider using Bing Search API.

The Bing Search API allows for 5000 queries a month, all of which you can choose to use on a single day itself. Again, this will solve your problem if your requirement is not a daily/regular one and you can look beyond Google.

Else, paying Google is your only way out.