Speed up my Rails Application which makes multiple web services HTTP requests using Typhoeus

Question

My current application makes around 30 parallel HTTP requests for getting all sorts of data from various website's APIs and also scrap some content from some websites.I am using Typhoeus for it as i have heard that it rapid for making HTTP requests.But i desperately want to speed up my application, I have done research for it and got suggestions for turning the Typhoeus get requests to head requests where ever possible.I also implemented techniques for eager loading, solved N+1 problems.But still i need to quicken my application a lot.

Any suggestions welcome.

Thanks in advance.

Answer 1

I've used resque before to handle a long running process in the background which greatly helped the site performance. If you are making those thirty parallel requests from the main process, it could slow things down quite a bit as I've read there are some issues with Ruby and threads.

You could try and put the scraping code into a worker process and then set your page refresh to see the updates.