Solution design for 26M records a year. Only 5000 are active at any time (Ruby on Rails and MySQL)

Question

I'm designing a booking system with Rails and MySQL.

Current business data indicates there will be 500,000 bookings a week. But only 5000 active bookings at any given time.

Once a booking is completed, it's falls into history where it can be queried for accounting purposes or diagnostics.

So, in a year we'll accumulate 26M bookings. But still only have a working set of 5,000 bookings. In real-time our system can perform very well, in fact most of the data could fit in RAM. But over time things will get painful as we accumulate data, so performance could be a problem.

I'd like to design this situation with minimal complexity and maximum performance.

Can anyone recommend an approach to handle this? My goal is to keep the solution structure simple and also to keep the dev-ops/deployment story simple?

Cheers

Tobin

Answer 1

I work in the telecommunication business for 10years, by now we process a average of 8MM calls a day, what give us a 40MM new records (average) every single day.

We use JAVA and Python with a Oracle for the main switch, and for processing and data analysis we use Hadoop and MapReduce.

We have 2 main servers with 64GB RAM, and 1TB SSD (RAID5) each, working in load balance and 2 backup servers, with 2 different networks, routing between 2 ciscos... easy approach with a solution that return everything fast.