Message queues

A message queue is an inter-process communication mechanism that enables different parts of a system to send and receive messages asynchronously.

Producers send messages to the queue and move on to other tasks, without waiting for consumers (aka. processors) to process the messages. Multiple consumers can pull messages from the queue, allowing work to be distributed and balanced across multiple nodes – supporting horizontal scaling.

Traditional message queues take in messages, queue them until they are ready to be processed, and then route them to consumers. (Message queues, like databases, therefore require durable storage.)

Message queues can be replicated for redundancy. Most message queues also support replication of consumers. Even if messages of a particular type are destined for one consumer, you can run multiple consumers of that type, and the queue will distribute the messages between them, effectively load balancing the processing.

Message queues can also be configured for message routing, eg. routing messages to different queues based on their content. A message broker is the broader middleware category that provides this routing, transformation, and pub/sub on top of queues.

Well known examples of message queues include RabbitMQ and ActiveMQ. Hosted solutions include Amazon SQS.

Traditional message queues are distinct from event buses, event streams, and stream processing systems like Kafka. Although event stream processing systems can be used as message queues, they are really designed for different use cases. Event-based systems tend to fan out messages to multiple consumers, whereas message queues tend to treat individual messages as being destined for one processor. Traditional message queues delete messages once they have been received by a consumer, whereas event stream processing systems retain messages for a period of time, allowing multiple consumers to process the same message.

Traditional message queues are designed for moderate data volumes, whereas modern event stream processing systems target high throughput and low latency.

Message queue use cases

Common use cases for message queues include sending emails, processing payments, and handling notifications. It is rarely important that producers get a response to these requests, so the calls can be asynchronous. A producer often only needs to know that an email has been queued for delivery, not wait for confirmation that it has been sent.

Anything that could be treated as a background job is a good candidate for a message queue. When the queue’s purpose is to execute computational units of work rather than route data between services, the higher-level abstraction is a task queue.

When a consumer repeatedly fails to process a message, the message can be diverted to a dead letter queue so that it does not block the primary queue or consume retry budget indefinitely.