Ruby's best way to process thousand async requests?

Question

The task is to process quote ticks almost realtime ( 1 second delay ).

At the moment I need to process about hundred quotes but it might increase to thousands. I'm looking at the EM currently to run ten threads each of which will run ten async requests and put data into some PubSub for later processing.

Also I'm testing Celluloid-IO with HTTP gem.

Am I on the right way? Tried to check GitHub for smth that I'm trying to implement but failed.

Answer 1

With the large amount of parallel requests sometimes its recommended to use Fibers instead of Threads as they are faster to fire and lighter in memory usage.

The most popular solution is EM-synchrony and you can use EM::Synchrony::Iterator for job scheduling and balancing.

Be aware that you should use according EM-libraries like "em-synchrony/em-http" for compatibility with EM-synchrony.

You can check this blogpost for more insights.