Good request from browser but bad request from ruby?

Question

I'm using the google custom search api and I'm trying to access it through some ruby code:

Here is a snippet of the code

req = Typhoeus::Request.new("https://www.googleapis.com/customsearch/v1?key={my_key}&cx=017576662512468239146:omuauf_lfve&q=" + keyword, followlocation: true)
res = req.run

It appears that the body of the answer is this one:

    <p>Your client has issued a malformed or illegal request.  <ins>That’s all we know.</ins>
'
    from /usr/local/lib/ruby/2.1.0/json/common.rb:155:in `parse'
    from main.rb:20:in `initialize'
    from main.rb:41:in `new'
    from main.rb:41:in `<main>'

When I try to do the same thing from the browser it works like a charm. Even more confusing is that this same code worked 12 hours ago. I only changed the keyword that it should look for, however it started returning the error.

Any suggestions? I'm sure that I have enough credits for more requests

Answer 1

You probably have problems with special characters in your get parameter keyword. If you enter the URL in your browser, the brower adjusts these. However, for ruby you need to escape these characters, in such a way that a string like "sky line" becomes "sky+line" and so on. There is a utility function CGI::escape, which is used like this:

require 'cgi'
CGI::escape("sky line")
=> "sky+line"

Your fixed code would look something like this:

req = Typhoeus::Request.new("https://www.googleapis.com/customsearch/v1?key={my_key}&cx=017576662512468239146:omuauf_lfve&q=" + CGI::escape(keyword), followlocation: true)
res = req.run

However, since you're using Typhoeus anyway, you should be able to use its params parameter and let Typhoeus handle the escaping:

req = Typhoeus::Request.new(
  "https://www.googleapis.com/customsearch/v1?&cx=017576662512468239146:omuauf_lfve",
   followlocation: true, 
   params: {q: keyword, key: my_key}
)
res = req.run

There's more examples on Typhoeus' GitHub page.