How an async response is usually getting back to the clent in typical reactive highload architecture?

Question

A typical highload reactive architecture is: (Q means queue)

And usually arrows are from client to DB at such sketches. My question is how the response is going back? The same queues and other stuff?

(Pictures links to presentations / video are appreciated)

Answer 1

In a system based on asynchronous messages (events), there is no concept of a synchronous “response”. However, some events might be related. Many message queues allow a message/event to have a correlation ID to determine how messages are related. For example:

1. A client publishes an event with ID 123.
2. A service receives the event 123, does some processing, and publishes an event with ID 456 and correlation ID 123.
3. The client receives the event 456. The client sees that this event was created in response to the previously sent event 123.

Whether the "response" event is published on the same queue or on a different message queue is an implementation detail.

While event-based systems are very flexible, they offer few guarantees. For example, we cannot generally ensure:

• that an event will be received exactly once
• that there will be exactly one response
• that a response will be received in a timely manner

This can make it difficult to interface asynchronous messages with synchronous protocols such as HTTP. It may therefore be necessary to account for the asynchronous nature at all levels of the design.

• For example, consider an event that will cause a irrevocable external action, such as sending an email. Due to technical limitations the same event may be received multiple times, but only one email should be sent per event. One possible solution is to store the processed event IDs in a transactional database. Events are discarded if they were already seen.

  This is still problematic if the email service is supposed to generate a confirmation event, but that confirmation gets lost. Then, re-sending the confirmation whenever the original event is re-received might be appropriate.

• As another example, consider a website that prepares some downloadable content on demand, but generating this content takes some time. The web server publishes an event that requests the download to be generated. What should the web server do in the meanwhile?

  One possibility is to wait, in the hope that there will be a response event within some timeout. However, this response is not guaranteed.

  Another alternative is to expose the asynchronous nature to the client, and respond immediately once the background processing has been triggered. (202 Accepted would be an appropriate status code). The HTTP response should offer some way to track the status of the request. Later, the backend service generates a response event. This could trigger some kind of notification to the client, whether that's a message through a websocket, an email, or simply the creation of a resource that the client occasionally polls.