What is Net::HTTP doing differently than executing curl in ruby?

Question

I have methods that GETs a resource from an external Rails server via JSON. One uses the net/http library, the other executes curl. They respond differently.

If I use net/http

def get_via_nethttp(endpoint, user, password, id)
  uri = URI.parse("#{endpoint}/items/#{id}.json")
  http = Net::HTTP.new(uri.host, uri.port)
  headers = {
    'Content-Type' => 'application/json',
    'Accept-Encoding' => 'gzip,deflate',
    'Accept' => 'application/json'
  }
  request = Net::HTTP::Get.new(uri.request_uri, headers)
  request.basic_auth(user, password)

  http.request(request)
end

then the return code is ASCII-8Bit encoded. Here's a snippet:

\x1F\x8B\b\x00\x00\x00\x00\x00\x00\x03\xBDWmo\xE28\x10\xFE+Vv\xA5...

However, if I use curl

def get_via_curl(endpoint, user, password, id)
  %x(curl -u #{user}:#{password} #{endpoint}/items/#{id}.json)
end

then a JSON string returns

"{\"id\":1234,\"title\":\"Foo\"}"

I've tried numerous ways to decode the output of net/http to no avail. In the end, I just want the JSON string, so I ended up going with curl. What's the difference that's causing this?

Answer 1

Setting in headers 'Accept-Encoding' => 'gzip,deflate' can result server sending you a chunked response. Just remove it.