message passing and signaling in distributed system
-
20-09-2019 - |
Question
I have a distributed video analysis system, which is composed of:
1. feature extraction: generated lots of features(20+) from each frame of the video
2. multiple detectors(in different machine):
* Each of them will get a subset of feature
* Each of them needs the features from multiple frames.
* Eg. Detector 1 needs feature 1-5 from 3 frames to start processing; Detector 2 needs feature 2-8 from 8 frames to start processing
My question is: how to do the communication between feature extraction block and multiple detectors, preferably in real-time? I have been looking at an event bus, but it's only for one process, will ZooKeeper in Hadoop be a better solution?
I am using Java. Any suggestion is welcomed.
Solution
In Java, you could consider using RabbitMQ message bus. There is a library with Java bindings.
RabbitMQ is based on a proven platform, offering exceptionally high reliability, availability and scalability along with good throughput and latency performance that is predictable and consistent. It has a compact, easily maintainable code base allowing rapid customisation and hot deployment. There are extensive facilities for management, monitoring, control and debugging and it is supported by a full range of commercial support services and an active community developing packages that extend the core system.