'Nesting too deep' error while retrieving JSON using HTTParty

Question

I am writing a very simple proxy wrapper for the reddit api so I can make cross domain JSONP requests (reddit does not allow JSONP calls to my knowledge).

I am using the HTTParty gem to make the call out and retrieve the response, which I immediately render out as json.

I am retrieving the subreddit json perfectly fine so far with this as my route:

  match "r/:name" => "api#subreddit"

... and this as my controller action:

def subreddit
    render :json => HTTParty.get("http://www.reddit.com/r/" + params[:name] + "/.json"), :callback => params[:callback]
end

As an example, the JSON that comes back is similar to this: http://www.reddit.com/r/pics/.json

My problem is when I attempt to grab the comments for a particular thread.

My route is as so:

match "r/:name/comments/:id" => "api#comments"

... and my controller action is this:

  def comments
    render :json => HTTParty.get("http://www.reddit.com/r/" + params[:name] + "/comments/" + params[:id] + "/.json"), :callback => params[:callback]    
  end

The JSON that I want back can be seen here: http://www.reddit.com/r/pics/comments/rhzgc/.json

When I call this action, however, I receive the following error message:

MultiJson::DecodeError in ApiController#comments
Nesting of 20 is too deep

The full stack trace follows:

json (1.6.6) lib/json/common.rb:148:in `parse'
json (1.6.6) lib/json/common.rb:148:in `parse'
multi_json (1.2.0) lib/multi_json/engines/json_common.rb:9:in `decode'
multi_json (1.2.0) lib/multi_json.rb:81:in `decode'
httparty (0.8.1) lib/httparty/parser.rb:116:in `json'
httparty (0.8.1) lib/httparty/parser.rb:136:in `parse_supported_format'
httparty (0.8.1) lib/httparty/parser.rb:103:in `parse'
httparty (0.8.1) lib/httparty/parser.rb:66:in `call'
httparty (0.8.1) lib/httparty/request.rb:222:in `parse_response'
httparty (0.8.1) lib/httparty/request.rb:192:in `handle_response'
httparty (0.8.1) lib/httparty/request.rb:75:in `perform'
httparty (0.8.1) lib/httparty.rb:391:in `perform_request'
httparty (0.8.1) lib/httparty.rb:343:in `get'
httparty (0.8.1) lib/httparty.rb:423:in `get'
app/controllers/api_controller.rb:12:in `comments'
actionpack (3.1.1) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (3.1.1) lib/abstract_controller/base.rb:167:in `process_action'
actionpack (3.1.1) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (3.1.1) lib/abstract_controller/callbacks.rb:18:in `block in process_action'
activesupport (3.1.1) lib/active_support/callbacks.rb:416:in `_run__2872109728488784816__process_action__2261783022290592822__callbacks'
activesupport (3.1.1) lib/active_support/callbacks.rb:386:in `_run_process_action_callbacks'
activesupport (3.1.1) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (3.1.1) lib/abstract_controller/callbacks.rb:17:in `process_action'
actionpack (3.1.1) lib/action_controller/metal/rescue.rb:17:in `process_action'
actionpack (3.1.1) lib/action_controller/metal/instrumentation.rb:30:in `block in process_action'
activesupport (3.1.1) lib/active_support/notifications.rb:53:in `block in instrument'
activesupport (3.1.1) lib/active_support/notifications/instrumenter.rb:21:in `instrument'
activesupport (3.1.1) lib/active_support/notifications.rb:53:in `instrument'
actionpack (3.1.1) lib/action_controller/metal/instrumentation.rb:29:in `process_action'
actionpack (3.1.1) lib/action_controller/metal/params_wrapper.rb:201:in `process_action'
activerecord (3.1.1) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (3.1.1) lib/abstract_controller/base.rb:121:in `process'
actionpack (3.1.1) lib/abstract_controller/rendering.rb:45:in `process'
actionpack (3.1.1) lib/action_controller/metal.rb:193:in `dispatch'
actionpack (3.1.1) lib/action_controller/metal/rack_delegation.rb:14:in `dispatch'
actionpack (3.1.1) lib/action_controller/metal.rb:236:in `block in action'
actionpack (3.1.1) lib/action_dispatch/routing/route_set.rb:65:in `call'
actionpack (3.1.1) lib/action_dispatch/routing/route_set.rb:65:in `dispatch'
actionpack (3.1.1) lib/action_dispatch/routing/route_set.rb:29:in `call'
rack-mount (0.8.3) lib/rack/mount/route_set.rb:152:in `block in call'
rack-mount (0.8.3) lib/rack/mount/code_generation.rb:96:in `block in recognize'
rack-mount (0.8.3) lib/rack/mount/code_generation.rb:89:in `optimized_each'
rack-mount (0.8.3) lib/rack/mount/code_generation.rb:95:in `recognize'
rack-mount (0.8.3) lib/rack/mount/route_set.rb:141:in `call'
actionpack (3.1.1) lib/action_dispatch/routing/route_set.rb:532:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/best_standards_support.rb:17:in `call'
rack (1.3.6) lib/rack/etag.rb:23:in `call'
rack (1.3.6) lib/rack/conditionalget.rb:25:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/head.rb:14:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/params_parser.rb:21:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/flash.rb:243:in `call'
rack (1.3.6) lib/rack/session/abstract/id.rb:195:in `context'
rack (1.3.6) lib/rack/session/abstract/id.rb:190:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/cookies.rb:331:in `call'
activerecord (3.1.1) lib/active_record/query_cache.rb:62:in `call'
activerecord (3.1.1) lib/active_record/connection_adapters/abstract/connection_pool.rb:477:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (3.1.1) lib/active_support/callbacks.rb:392:in `_run_call_callbacks'
activesupport (3.1.1) lib/active_support/callbacks.rb:81:in `run_callbacks'
actionpack (3.1.1) lib/action_dispatch/middleware/callbacks.rb:28:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/reloader.rb:68:in `call'
rack (1.3.6) lib/rack/sendfile.rb:101:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/remote_ip.rb:48:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/show_exceptions.rb:47:in `call'
railties (3.1.1) lib/rails/rack/logger.rb:13:in `call'
rack (1.3.6) lib/rack/methodoverride.rb:24:in `call'
rack (1.3.6) lib/rack/runtime.rb:17:in `call'
activesupport (3.1.1) lib/active_support/cache/strategy/local_cache.rb:72:in `call'
rack (1.3.6) lib/rack/lock.rb:15:in `call'
actionpack (3.1.1) lib/action_dispatch/middleware/static.rb:53:in `call'
railties (3.1.1) lib/rails/engine.rb:456:in `call'
railties (3.1.1) lib/rails/rack/content_length.rb:16:in `call'
railties (3.1.1) lib/rails/rack/log_tailer.rb:14:in `call'
rack (1.3.6) lib/rack/handler/webrick.rb:59:in `service'
/Users/emcummings/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/webrick/httpserver.rb:111:in `service'
/Users/emcummings/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/webrick/httpserver.rb:70:in `run'
/Users/emcummings/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/webrick/server.rb:183:in `block in start_thread'

What am I doing wrong here? If the JSON is too big to parse (I find that sort of hard to believe!) what can I do?

EDIT:

Looks like dropping back to the standard NET::HTTP lib solved my problem:

uri = URI.parse("http://www.reddit.com/r/pics/comments/rhzgc/.json")    
response = Net::HTTP.get_response(uri)
render :json => response.body, :callback => params[:callback]
#OK!

... but I still am curious why I am failing this same call with HTTParty!

Thanks for the help!

Answer 1

It's not that the JSON is too large. It's nested too deeply. HTTParty tries to decode the results that it gets automatically. Following your stack trace, and the HTTParty dependencies, it relies on multi_json which is using the json gem.

Inside of json there is lib/json/pure/parser.rb. The default max depth set is set in there, specifically on line 79. Something in your returned JSON is 20+ levels deep, triggering the exception.

 if !opts.key?(:max_nesting) # defaults to 19
   @max_nesting = 19

Answer 2

Switched to the OJ JSON parser by adding the following to my Gemfile:

gem 'oj'

and this issue resolved itself.

Answer 3

It looks like this can be worked around by calling to_json with the max_nesting option set.

json = obj.to_json(max_nesting: 50)