TS-23: Messages and Events

This technical standard covers best practices for designing and implementing messages and events in message-driven architectures (MDA).

The focus of this standard is asynchronous communication between services within a single organization’s internal network ("intra-organization communication"). See also TS-22: Webhooks. Webhooks are a subset of message-driven communication patterns, in which event-oriented messages are exchanged via HTTP between systems owned and operated by different parties ("inter-organization communication"). Intra-organization and inter-organization messages serve different purposes and have different sets of constraints, and therefore their designs can look very different. Nevertheless, there is some overlap between the two, and so some aspects of both technical standards will inevitably cross over each other, too.

Message systems

When it comes to implementing message-driven communication within an internal network, there are many options to choose from. The following are some common architectural patterns for message-driven communication.

• Message broker pattern: A central message broker routes messages between producers and consumers. Examples include RabbitMQ, ActiveMQ, and Kafka.
• Event bus pattern: A lightweight event bus distributes messages to multiple subscribers. Examples include Redis, NATS, and internal event buses.
• Publish-subscribe pattern: A one-to-many message distribution pattern where messages are published to a topic and multiple subscribers can receive them. This is supported by MQTT, AMQP, Kafka, and Redis.
• Request-reply pattern: A synchronous-style communication pattern, implemented over async transport, where a request is sent and a reply is expected. This is supported by most message queues.
• Remote Procedure Call (RPC) pattern: Synchronous communication between services, where one service calls methods on another service as if they were local. Examples include gRPC and Apache Thrift.

Selection criteria for choosing a message transport protocol and delivery mechanism include:

• Latency versus throughput: Low-latency protocols like gRPC and NATS are suitable for real-time applications, while something like Kafka is better for high-throughput scenarios.
• Delivery guarantees: Protocols like AMQP and Kafka offer strong delivery guarantees, while MQTT provides different levels of Quality of Service (QoS), each level adjusting the trade-offs between reliability and performance.
• Message ordering: Kafka’s partitioning model supports ordered message delivery, while AMQP queues can also provide ordering guarantees.
• Scalability: Kafka is designed for massive scale, while RabbitMQ and NATS are for moderate scale.
• Complexity: Kafka and AMQP are feature rich but more complex for it, while MQTT and NATS focus on simplicity.

The most common convention in modern service-oriented architectures is to use either message brokers (RabbitMQ/AMQP) for traditional messaging patterns or event streaming platforms (Kafka) for event-driven architectures, with gRPC increasingly popular for synchronous service-to-service communication where the lowest possible latency is required.

Message queue protocols

A number of open protocols exist for message queues, each with its own strengths and weaknesses. The most widely used message queue protocols are:

• AMQP (Advanced Message Queuing Protocol): A binary protocol that runs over TCP. It is widely supported by message brokers such as RabbitMQ, ActiveMQ, and Azure Service Bus. AMQP supports complex routing, transactions, and delivery guarantees. It is best suited for enterprise messaging scenarios that require reliable delivery and complex workflows.
• MQTT (Message Queuing Telemetry Transport): A lightweight protocol that runs over TCP, and also supports WebSockets to allow clients to be web applications that run in web browsers. MQTT supports a publish-subscribe model with QoS levels. It is designed for resource-constrained environments (eg. low-bandwidth, high-latency networks), and it is widely used in IoT applications and telemetry for this reason. This protocol is used by Mosquitto, HiveMQ, AWS IoT Core, and others.
• STOMP (Simple Text Oriented Messaging Protocol): A text-based protocol that runs over TCP. It is a simpler alternative to AMQP that is designed to be easy to implement and debug. STOMP is popular in simple messaging scenarios implemented through low-level scripting languages.

Proprietary messaging systems

In addition to open protocols, there are also many proprietary messaging systems that offer their own protocols and delivery mechanisms. Some of the most popular proprietary messaging systems include:

• Apache Kafka: A distributed streaming platform that uses its own binary protocol. Kafka is designed for high-throughput, low-latency event streaming. It is best suited for event sourcing, log aggregation, and real-time data pipelines.
• NATS: A lightweight messaging system that uses its own protocol. NATS is designed for simplicity and speed, with a focus on simple pub-sub messaging. It is a popular choice in microservices and cloud-native applications.
• Redis Pub/Sub and Streams: Redis is an in-memory data store that includes a simple pub-sub messaging system with optional persistence. Redis uses its own RESP protocol. It is best suited for fast, simple messaging and caching integration.

Platform-specific solutions

Public cloud service providers offer their own managed messaging services, which may be a good choice if you are already using that cloud provider’s infrastructure. Examples include:

• Amazon SQS/SNS: AWS-managed queuing and pub-sub services over HTTP.
• Azure Service Bus: Microsoft-managed messaging service that supports AMQP, HTTP, and proprietary protocols.
• Google Cloud Pub/Sub: GCP-managed messaging service that runs over HTTP or gRPC.

Remote Procedure Call (RPC) frameworks

RPC frameworks enable synchronous communication between services, allowing one service to call methods on another service as if they were local. Though not strictly "messaging systems" – messaging is an asynchronous communication pattern – it is worth mentioning RPC frameworks here as they are commonly used for inter-service communication, running alongside message queues and event streams in service-oriented systems. RPC is more suitable for time-sensitive queries and commands, where low latency and immediate responses are required.

Some popular RPC frameworks include:

• gRPC: A high-performance RPC framework developed by Google. It uses HTTP/2 as the transport protocol and Protocol Buffers for serialization. gRPC supports bi-directional streaming and is well-suited for microservices, polyglot environments, and low-latency APIs.
• Apache Thrift: An RPC framework developed by Facebook. It supports multiple transport protocols (eg., TCP, HTTP) and serialization formats (eg., JSON, binary). Thrift is best suited for multi-language service integration.

HTTP APIs

Besides the specialist message systems described above, internal HTTP APIs are also commonly used for communication within internal networks. While conventional HTTP endpoints support synchronous polling, push notifications can be implemented using webhooks (aka. HTTP callbacks), Server-Sent Events (SSE), or WebSockets – allowing for true asynchronous message communication patterns.

Internal HTTP APIs can be a good choice for simple integrations, where fast real-time responses are not critical, and otherwise where the overhead of setting up and maintaining a specialist messaging framework is not justified.

Typically, do-it-yourself HTTP-based messaging systems use HTTPS for transport and message semantics, and JSON for data interchange. But you have full freedom to design your own HTTP message conventions. Industry conventions such as Standard Webhooks and CloudEvents can provide some useful guidance, though for internal communication you have flexibility to design a proprietary system that best fits your own requirements.

The remainder of this section covers RECOMMENDATIONS for the design and implementation of internal HTTP APIs and internal webhooks. Aspects of TS-21: HTTP APIs and TS-22: Webhooks are also relevant here; this section is extended guidance that is specific to HTTP messaging within internal networks, rather than over the public internet.

Secure protocol

Internal networks are not inherently secure. Therefore HTTPS is REQUIRED to encrypt messages in transit, to protect against eavesdropping and man-in-the-middle attacks.

Data interchange formats

It is RECOMMENDED to use JSON as the data interchange format for HTTP messages, due to its widespread adoption, human readability, and compatibility with all mainstream programming languages and platforms.

Other formats, such as XML or Protocol Buffers, may be used in specific scenarios where their features are desired. But JSON SHOULD be the default go-to choice for asynchronous communication between most services.

Message schema

Designing a robust, flexible, scalable, and maintainable message schema is a crucial aspect of the design of distributed systems with asynchronous message-based communication patterns. The message schema defines the structure of data exchanged between different components and services within a distributed system.

Some message systems will impose their own opinions on message schema design, but when using internal HTTP APIs you have full freedom to design your own message schema.

Ideally, all asynchronous communication between nodes within a distributed system should use a consistent, versioned JSON schema for all types of messages. Ideally, synchronous service-to-service communication should also use the same message schema, to maximize consistency across all communications. Standardization on the high-level design of all messages across an entire system reduces overall complexity, encourages code reuse via shared libraries, and improves interoperability between services.

Therefore, it is RECOMMENDED to model all types of messages using a unified schema. This means defining an extensible structure that can scale to represent all kinds of messages.

Message types

Broadly, there are three categories of messages: events, commands, and queries.

• Events represent things that have happened in the service emitting the event (eg., user.created, order.placed).
• Commands represent requests for operations to be performed by other services (eg. sendEmail, refundOrder).
• Queries are requests for data (eg., getUserDetails, listOrders).

All three message types are closely related. The difference is mostly in the statement of intent that underpins their semantics: whether the message is telling another component to do something (a command), whether it is requesting data (a query), or whether it is informing other components that something has happened (an event).

Queries can be thought of a sub-type of commands. They differ from commands in that they are read-only; they are not expected to change state, whereas commands may do so.

Commands and queries will typically spawn one or more new events that inform other components of the results of executing the command, or the results of the query. Thus, a cascade of events may be triggered by a single initial command or query.

In most cases, it will be expected that producers of queries will also be consumers of subsequent events that return the requested data asynchronously.

All three types of message are used in message-driven architectures. A good message schema will accommodate all three types of message in a consistent way.

Message body

There are two parts to a message schema: the message’s main payload, and metadata for the message container. These two parts SHOULD be clearly differentiated in the schema.

The following high-level design achieves this separation by placing the payload inside a data field, with other fields at the top-level of the data structure capturing metadata.

{
  "spec_version": "string",  // Message schema version number.

  // Metadata:
  "message_id": "string",    // Unique identifier for the message.
  "created_at": "string",    // Time of message creation, RFC 3339/ISO 8601 format
  "type": "string",          // One of: "event", "command", "query".
  "name": "string",          // Name of the event, command, or query.

  "data": {
    // Payload:
    "field1": <value>,       // Payload schema is specific to each
    "field2": <value>,       //   type of event, command, and query.
    "field3": <value>
  }
}

An alternative design would be to move the message metadata into the HTTP headers, leaving just the payload in the message body. This approach has some advantages, such as better separation of concerns and easier access to metadata for routing and processing. However, the main disadvantage of this alternative design is that the message payload becomes less portable. It is tied to the HTTP protocol, making it harder to reuse the same message schema across different transport protocols.

It is RECOMMENDED that consumers implement validation of incoming messages against a schema. The following JSON Schema can be used to validate messages that conform to the high-level design shown above.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",

  "type": "object",
  "properties": {
    "spec_version": {
      "type": "string"
    },
    "message_id": {
      "type": "string"
    },
    "created_at": {
      "type": "string"
    },
    "type": {
      "type": "string"
    },
    "name": {
      "type": "string"
    },
    "data": {
      "type": "object",
      "patternProperties": {
        "[a-z][a-zA-Z0-9_.]*$": {
          "type": ["string", "number", "boolean", "object"]
        }
      },
      "additionalProperties": false
    }
  },
  "required": [
    "spec_version",
    "message_id",
    "created_at",
    "type",
    "name",
    "data"
  ],
  "additionalProperties": true
}

Metadata

The metadata fields capture all the essential information needed to support the tracking and processing of messages.

Besides the recommended fields shown in the example above, other metadata fields MAY be included as needed, such as source and correlation_id (for tracing). The metadata fields MUST be chosen carefully, to accommodate changing metadata requirements over time.

Spec version

The spec_version field indicates the version of the message schema. Consumers can use the spec_version field to differentiate their processing of messages encoded to different schema versions.

Transitions to new schema versions SHOULD be done incrementally. This is done by having producers emit duplicate messages in both the old and new schema versions for a period of time, while consumers are migrated to the new schema. This process allows breaking changes to be introduced to schema designs if required. But better to evolve event schema in a non-breaking way wherever possible.

Message schema versioning SHOULD follow semantic versioning principles. See TS-11: Versioning.

Message schema SHOULD evolve separately to the public API of the service producing the events. Therefore message schema versioning SHOULD be independent of API versioning. See also TS-21: HTTP APIs.

Message ID

The message_id value serves as an idempotency key, allowing consumers to safely process duplicate messages. It SHOULD be a UUID. In turn, this supports retries and other mechanisms that improve the reliability of message delivery.

Message type and name

The value of the type field indicates whether the message is an event, command, or query. The value of the name field indicates the specific name of the event, command, or query.

Events, commands, and queries MAY each have different naming conventions. For example, events may use dot-noted event names like user.created and invoice.paid, in which the first part identifies an entity type and the second part identifies a type of mutation. Meanwhile, commands and queries may use camelCase names like sendEmail and getUserDetails.

All the possible names of events, commands, and queries make up a catalog of message types. The message catalog documents all the possible events, commands, and queries that a system may communicate internally via messages.

Prefer to design a large catalog of granular message types. Each type of message should align with a very specific use case. At the same time, don’t fragment unnecessarily, such that subscribers need to reconstruct discrete state changes from multiple disparate messages.

Timestamp

The created_at field captures the time at which the message was created. The timestamp SHOULD be in RFC 3339/ISO 8601 format, and in the UTC timezone – as per TS-47: Dates and Times.

This value MUST NOT change when messages are retried or redelivered. It captures the original creation time of the message.

Including this field allows consumers to understand the timing of events, commands, and queries. Since it is not possible to guarantee that messages are delivered to consumers in the same order in which they were created, the created_at timestamp allows consumers to make sure they don’t process messages out of order.

If it is important that consumers process messages, not only in the right order, but also without skipping any messages in between, then additional mechanisms are needed to enforce this. It is RECOMMENDED to include a sequence field in the metadata, which captures an integer that increments by one for each new message created in a sequence. This allows consumers to detect and handle any gaps in the sequence of messages they receive.

Since messages may be dropped or delayed, for example due to network issues, there are inherent limitations to the guarantees that can be made about message ordering and delivery. See Delivery and reliability, below, for guidance on managing this.

The created_at field can also be used by consumers to protect themselves from replay attacks. See Authentication and security, below, for more information.

Payload

The data field contains the main payload of the message. The structure of the payload is specific to each type of event, command, and query. Each message type+name should have a well-defined payload schema.

Payloads MUST be composed from a global library of common data types and structures, for maximum consistency and reusability. For example, if multiple events include user information, then they SHOULD all use the same User data structure.

The size of event payloads can impact delivery reliability and performance. Therefore, try to keep payloads small – under 1MB – and focused on the essential data needed by consumers. Consider opening new API endpoints from which event consumers can fetch additional information about the events they receive, if needed.

Authentication

It is RECOMMENDED to use a message-level authentication system to verify the authenticity and integrity of messages. The transport-level security provided by HTTPS (TLS) is not sufficient on its own, since messages may be intercepted and modified by malicious actors within the internal network.

The most common authentication pattern in message-based communication is HMAC (Hash-based Message Authentication Code) with SHA-256 hashing – a symmetric key algorithm. Other options include asymmetric signatures generated and verified by public/private key pairs, and bearer token authentication (eg., JWT). Bearer tokens can be a good choice where it is desirable to encode claims and scopes (ie. permissions) in messages. Basic authentication is not recommended, since its security depends entirely on end-to-end transport encryption (TLS/HTTPS). More advanced options such as OAuth and mutual TLS are generally not appropriate for internal message communication, due to their complexity and operational overhead.

See TS-22: Webhooks for an overview of all the options for authenticating HTTP messages, and the trade-offs of each approach. The default option RECOMMENDED by this technical standard is HMAC signatures with SHA-256 hashing. A scheme similar to that described in TS-22, and based on Standard Webhooks, is RECOMMENDED to protect messages from tampering, reducing susceptibility to replay attacks and other threats.

Signatures SHOULD be base64-encoded for compactness in transit.

POST /message HTTP/1.1
Host: 172.22.104.11
Content-Type: application/json
X-Message-ID: 550e8400-e29b-41d4-a716-446655440000
X-Message-DateTime: 2024-10-01T12:34:56Z
X-Message-Signature: sha256=a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3

{
  // ...
}

IP allow-listing MUST NOT be depended upon for authentication, but it MAY be used in addition to authentication, to provide an extra layer of security. See TS-22: Webhooks for more information.

Delivery and reliability

It is never possible to guarantee delivery of messages, or the correct sequencing of messages, between nodes within distributed systems. Messages may be dropped or delayed, for example due to network issues.

The following guidelines help to design systems that can handle the inherent unreliability of message delivery. These guidelines are protocol-agnostic, and apply equally to all types of message delivery systems, including message brokers, event buses, publish-subscribe systems, and HTTP-based messaging systems.

See also TS-22: Webhooks, which retreads some of this ground.

Idempotency

It is strongly RECOMMENDED that messages be designed to be idempotent. This means that the same message can be resent to a consumer multiple times without causing unintended side effects. Idempotency is crucial for ensuring that consumers can safely process duplicate messages, which may occur due to retries or network issues.

To achieve idempotency, messages should include a unique identifier, such as a message_id field. Producers MUST generate a unique message_id for each message they send, and this MUST be unique (a UUID is recommended). Consumers then have the option of logging this identifier so, if they receive the same message again, they can recognize it as a duplicate and ignore it.

Timeouts

Message delivery systems MUST implement reasonable timeout values for message deliveries. Timeout values SHOULD typically be between 10 and 30 seconds. After the timeout has elapsed, the message delivery is marked as failed and enters the retry system.

Retries and circuit breakers

It is RECOMMENDED to implement retry logic with exponential backoff plus jitter for failed deliveries. A common pattern is immediate retry, then delays of 1min, 5min, 30min, 2hrs, and 8hrs, before giving up and marking the message’s delivery as failed. Adjust the intervals based on the time-sensitivity of each message; shorter initial intervals may be appropriate for some use cases.

It is RECOMMENDED to add random jitter to retry intervals. When multiple clients experience failures simultaneously – which will be the case for a service outage – they may all retry at the same intervals:

• Client A: Retry at 30sec, 1min, 2min, 5min…​
• Client B: Retry at 30sec, 1min, 2min, 5min…​
• Client C: Retry at 30sec, 1min, 2min, 5min…​

The effect is "retry storms" or the "thunder herd" problem, in which lots of retries are seemingly synchronized across multiple clients, enough to overwhelm a service while it is still recovering from failure, causing new failures. Adding randomness (jitter) to retry intervals helps to spread out retry requests more evenly over time.

• Client A: Retry at 38s, 1min 15sec, 2min 5sec, 5min 35sec…​
• Client B: Retry at 42s, 1min 5sec, 2min 25sec, 4min 50sec…​
• Client C: Retry at 25s, 59sec, 2min 10sec, 5min 10sec…​

Producers MUST implement sensible defaults for retry intervals. In some cases it will be desirable to customize retry intervals for different types of message. Consumers SHOULD be able to configure the retry intervals for messages sent to them, overriding the defaults.

In addition, consumers MUST be able to retrieve their "dead letters" (messages that could not be delivered after multiple retries). This will typically involve consumers requesting a replay of failed messages, via an API endpoint (or dashboard for human users). Alternatively, dead letters could be saved to a log, from which consumers can retrieve them as a collection, to reconcile their synchronized state.

Producers MUST implement circuit breakers to temporarily stop deliveries to consistently-failing endpoints, to avoid overwhelming them. As with retry intervals, circuit breaker timeouts SHOULD be configurable by consumers, to accommodate different failure-recovery characteristics of different systems.

Rate limiting

Consumers MAY implement rate limiting on incoming messages. Doing so helps to protect against vulnerabilities such as denial-of-service (DoS) attacks, it helps to manage overall load on consumer systems.

Consumers MAY use the Retry-After header to inform producers when they are being rate limited, and when the producer can resume sending messages. Producers SHOULD respect the Retry-After header, using the information to customize the retry intervals for messages sent to that consumer.

Statelessness

Where possible, producers SHOULD design events to be stateless. This means that each message is self-contained; it includes all the information needed for the consumer to process it, without relying on any external state or context.

This is especially beneficial when the processing of events is not dependent upon the processing of prior events – since there can be no guarantees that those prior events will have been received or processed successfully.

Therefore, it is RECOMMENDED to avoid using sequence fields in event metadata, and not to require consumers to reconstruct state from the full sequence of events, processed in the right order without gaps.

An alternative design is to transmit no state at all in events. Such messages would not have payloads. These events are simply notifications that something has changed in the publisher service. Consumers are required to synchronize their state by making regular requests to API endpoints, in response to those notifications.

There are pros and cons to both approaches. "Thin" payloads tends to be beneficial in webhooks in public HTTP APIs, while "fat" payloads tend to be more appropriate for internal messaging systems. The right balance depends on the specific use case.

Service level agreements

Retries, timeouts, rate limiting, and other such policies for message delivery MUST be clearly defined in service level agreements (SLAs).

Response codes

When messages are delivered over HTTP, the consumer MUST return an HTTP status code that indicates whether the message was received and processed successfully. The producer uses this response to decide whether to retry delivery.

• 200 OK — The message was received and processed successfully. The producer SHOULD NOT retry.
• 202 Accepted — The message was received and queued for processing, but processing has not yet completed. This is the RECOMMENDED response for asynchronous processing. The producer SHOULD NOT retry.
• 4xx (client error) — The message was rejected due to a problem with the message itself (eg. malformed payload, schema validation failure, authentication failure). The producer SHOULD NOT retry, since retrying the same message will produce the same error. The consumer SHOULD include a descriptive error message in the response body.
• 5xx (server error) — The consumer encountered an error while processing the message. The producer SHOULD retry, following the retry logic described above.

A 2xx response acknowledges receipt; it does not guarantee that processing has completed. For at-least-once delivery semantics, the producer MUST treat any non-2xx response as a failure and retry.

Documentation

Message publishers MUST provide comprehensive documentation to support the integration of messages in consumer systems.

Documentation MUST include a full message catalog of all events, commands, and queries emitted by each service. Large message catalogs SHOULD be easily searchable. Event Catalog is an open-source tool that can be used to create and maintain message catalogs.

Message schemas MAY be documented using JSON Schema. JSON Schema is RECOMMENDED for internal message schemas because it allows messages and events to evolve independently from the request-response API lifecycle. A message schema documented in JSON Schema can be versioned and distributed without coupling to a particular API description format.

Interface description languages (IDLs) such as AsyncAPI and OpenAPI tend to be more appropriate for public APIs and webhooks. OpenAPI v3.1 introduced support for describing webhook event schemas, making it possible to document webhook delivery contracts alongside a conventional request-response API. But for internal message-driven systems, where messages evolve on their own lifecycle, a standalone JSON Schema is the more flexible choice.

References

• Standard Webhooks is an initiative to document common conventions for message delivery. The guidelines target webhooks – ie. event-oriented messages transmitted over the public internet between organizations – though there are many good ideas here that are applicable to internal communication design, too. Standard Webhooks is a catalog of the most common patterns seen in the webhook implementations of major public web services, and it specifies a standard based on those patterns and emerging best practices. It captures conventions for event naming, payload structure, security (signatures), and delivery patterns.
• CloudEvents is an effort by the Cloud Native Computing Foundation to standardize event schemas. It’s focus is on improving interoperability across different cloud providers and platforms. It does this by specifying a generic specification for event data and metadata that can be mapped to a wide variety of messaging and transport protocols and message encoding formats. It’s more opinionated than Standard Webhooks, and its focused exclusively on message schema. As a protocol-agnostic standard, it does not cover topics such as security and authentication.
• Shared Signals and Events (SSE) is an OpenID Foundation initiative that is developing standards and best practices for the secure, privacy-protected transmission of messages and events over the public internet.