Data transfer between servers

Question

Assuming you have two bunch of servers, the first one dedicated to front and the other one, dedicated to process information from the fronts. What is the best way to transfer data from the fronts to the process servers;

I tried different techs on small amount of data:

• tried to dump data into files and retrieve them from the process servers... that's ok very secure because you never lose your data, but it uses a lot of disc write capacity.

• also tried sockets very cool

But sincerely I still don't know what is the best way to treat a huge data stream between servers.

Can someone point me in the right direction?

Answer 1

I would say the best option is to use a persistent queue, like RabbitMQ. That way if the receiving servers go down then your transfer is not lost and the transfer will simply continue when the receiving server pulls the data off the queue