TS-20: Network APIs

This technical standard defines broad guidelines for network APIs of all kinds – REST, GraphQL, gRPC, etc. The emphasis here is on considerations such as security and reliability, and especially performance optimization.

A network API is defined as a service interface exposed over a network using a standard protocol – such as HTTP, gRPC, or WebSocket – that allows clients to communicate with the service across process boundaries and potentially across machines. Network APIs introduce latency, potential failures, and security concerns that must be explicitly handled.

Network APIs are distinct from in-process APIs. An in-process API is defined as a service interface (typically a library or module) that is called directly within the same process. Calls are normally synchronous and failures immediate.

This technical standard also covers guidelines for implementing abstractions for network APIs in application code.

Inter-service communication patterns

In general, there are three ways to implement inter-service communication in distributed systems:

• Commands
• Messages
• Events

Each approach trades off coupling and decoupling. Command-driven systems are tightly coupled but simple, while event-driven systems achieve the greatest decoupling at the cost of added complexity.

In practice, most distributed systems use a mix of all three styles. You will see commands used for synchronous request-response patterns (especially at system boundaries), and events for asynchronous fire-and-forget patterns. Communication from the outside world tends to be RPC-style, and gateway services typically make blocking RPC-style requests to core services.

RPC-style communication

RPC-style communication involves direct, synchronous calls from one service to another. The calling service initiates a request, waits for a response, and then continues execution.

This is the most straightforward approach to inter-service communication, and it is familiar to most developers – it mirrors typical function calls within a single process.

RPC-style communication is appropriate for request-response interactions where the calling service needs an immediate answer. It is commonly used at system boundaries (between client and API gateway) and for operations that are inherently synchronous in nature.

The big advantage of this communication pattern is its simplicity. The control flow is explicit and easy to understand.

The trade-off is that RPC-style communication creates tight coupling between services. The calling service depends on the availability of the called service, and failures propagate directly. Network latency is exposed to the caller, and the blocking nature means resources are held while waiting for responses.

For these reasons, RPC-style communication should be used judiciously in internal inter-service communication, reserved primarily for gateway-to-service boundaries and where the coupling is acceptable.

RPC communication can also be asynchronous, where the calling service does not wait for a response. This is suitable where the caller does not require an immediate answer (or needs no response at all).Asynchronous RPC improves decoupling compared to synchronous RPC, since the caller is no longer blocked waiting for a response.

However, asynchrmonous RPC patterns introduce accidental complexity into the calling service. The caller must now handle concerns like retries, timeouts, and error handling for operations that may fail.

An improvement on this design is to redirect asynchronous requests through a message broker. The calling service sends the request to the intermediate component, the broker, which takes responsibility for reliable delivery, retries, and error handling. This extracts all that non-essential complexity from the application code and moves it to dedicated infrastructure. This is the essence of message-driven communication…​

Message-driven communication

Message-driven communication is similar to RPC-style in intent – one service instructs another to perform a specific action – but the communication is now asynchronous and indirect. Instead of making direct calls, services send commands to a central message broker, which routes those messages to the appropriate consumers.

This decoupling of direct dependencies is a key advantage over RPC-style communication. Because the calling service doesn’t call the target service directly, the target service can be temporarily unavailable without immediately failing the caller. The message is persisted in the broker, and the broker can keep retrying delivery to the target service until the message is successfully accepted. This improves resilience compared to synchronous RPC calls.

Message-driven systems are also inherently asynchronous. The sending service doesn’t wait for the target service to complete. This allows better resource utilization and higher throughput compared to blocking synchronous RPC requests.

However, message-driven communication remains fundamentally imperative. One service is commanding another to do something. The sender has a specific intent about what the receiver should do, which means services remain semantically coupled, even if they are decoupled in location and availability.

Message-driven communication sits between RPC-style and event-driven on the coupling spectrum. It is useful for scenarios where you need the resilience and asynchronous benefits of a message broker, but the communication is still inherently command-oriented.

However, we should prefer event-driven patterns where possible, as they provide the ultimate decoupling and the greatest flexibility for system evolution.

Event-driven communication

In event-driven architectures, instead of one piece of code commanding another to do something, it tells other parts of the system about interesting things that have happened. Other parts of the code may or may not be listening for those event notifications, and they can choose to act on them or ignore them.

The event publisher does not expect responses from any other part of the system.

This is the fundamental pattern used in all modern GUI systems, and also in JavaScript event loops. The pattern scales to distributed systems through messaging infrastructure.

In distributed systems, event broadcasts are self-contained, atomic messages. The routing and lifecycle management of those event messages is delegated to dedicated infrastructure components, such as an event bus or a message broker, which take care of infrastructure concerns like scaling, resilience, and persistence. This leaves application code (the services) free to specialize in the problem domain.

The secret to well-designed event-driven systems is for services to model the problem domain, not technical concerns. Services should model the essential complexity of the domain, while the events infrastructure removes much of the accidental complexity that would otherwise exist in the those services.

The sign of a well-designed distributed system is when all accidental complexity is moved into the infrastructure, leaving service code to focus purely on domain modeling.

The messaging system also provides a convenient, centralized place to monitor inter-service communication, inject observability, and implement cross-cutting concerns.

The event objects themselves should model real-world processes and carry business meaning: orders placed, subscriptions cancelled, etc. Be wary of having too many technical or tactical events.

When designed this way, event-driven architecture gives systems more flexibility to grow in unforeseen ways. For example, to add new processing on customer orders, you can simply add a new service and have it listen to the relevant "order" events, without modifying existing services. This isolates change and allows systems to grow incrementally, sometimes in unplanned ways.

Event-driven architecture, done well, can significantly reduce the complexity of otherwise very complex systems.

Abstracting network APIs in application code

In application code, network APIs SHOULD NOT be overly abstracted such that the network is hidden from application developers.

One of the dangers of abstracting network calls behind "in memory" object calls, for example, is that it can lead to the false assumption that network calls are instantaneous. (The assertion that "latency is zero" is the second of the famous fallacies of distributed computing.)

When network calls are abstracted too much, it can lead to a lack of awareness about the underlying network operations, and of their potential latencies and failure modes. This in turn can lead to application logic that does not account for network latency, that wastes bandwidth with unbounded payloads, or that does not gracefully handle network failures.

For distributed systems to be fault tolerant and robust, applications are required to implement patterns such as timeouts, retries, and fallbacks.

So, it is better practice to design application code to reflect the underlying network operations and their potential latencies, even if this means more verbose application code.

Aim for network transparency in application code, meaning that network operations are explicit and visible in the codebase.

Performance optimization

Performance optimization is a key consideration in network API design, with techniques spanning multiple layers. These include implementing caching, using content delivery networks, balancing load across multiple servers, and using bandwidth-efficient data formats.

Minimizing payload sizes through compression, managing the balance between over-fetching and under-fetching, reusing connections through connection pooling, and using asynchronous processing all contribute to better performance.

Additionally, offloading routing and security concerns to an API gateway can simplify the core API implementation.

Target metrics

Performance targets should be based on the nature of the operation. Critical operations should aim for response times of 200ms or less, standard operations 500ms or less, and complex operations 2000ms or less.

Client timeouts should be set to 3x the target response time.

For payload sizes, general APIs should stay within 10MB while file upload endpoints can accommodate up to 100MB.

Payload optimization

Reducing payload size requires a multi-faceted approach.

Implement pagination for collections that exceed 100 items by default to avoid overwhelming clients with large responses.

Enable gzip or brotli compression for payloads larger than 1KB. For large datasets and real-time data, use streaming with chunked responses rather than sending monolithic payloads.

Support field filtering (eg. ?fields=id,name,email) to prevent over-fetching data that clients don’t need. Provide batch endpoints to allow clients to fetch multiple resources in a single request, reducing under-fetching and the number of round trips required.

Finally, support partial updates using HTTP PATCH operations or equivalent mechanisms in other protocols.

Caching strategies

Caching strategies SHOULD be implemented on both sides of a network API: upstream on the server-side for back-end performance, and downstream on the client-side for reduced network traffic.

Pay careful attention to cache invalidation and purging strategies to ensure that stale data doesn’t cause problems.

See TS-21: HTTP APIs for more guidance on caching strategies.

Reliability and resilience

Building reliable and resilient APIs requires attention to error handling, rate limiting, and failure recovery patterns.

Error handling

Error responses must be standardized across all network APIs.

Include actionable error messages that provide enough context for clients to understand what went wrong, paired with documented error codes that can be reliably parsed.

Use consistent response status codes throughout your API. For HTTP APIs, this typically means using 200, 400, 401, 403, 404, 429, 500, and 503 with clear semantics for each.

In batch operations, where some requests succeed and others fail, report partial successes so clients understand exactly which operations succeeded and which ones need to be retried.

Rate limiting and backoff

Apply rate limiting at the appropriate level for your use case. This might be per client, per endpoint, or per resource depending on your needs.

Include rate limit information in response headers, where possible, so clients can see how close they are to hitting limits and adjust their behavior accordingly. When clients do hit rate limits or encounter transient failures, they should implement exponential backoff for their retry logic.

It’s important to distinguish between rate limits (temporary restrictions to protect the API) and usage quotas (permanent restrictions on what a customer is allowed to do), as they have different semantics and should be communicated differently to clients.

Circuit breakers

The circuit breaker pattern helps prevent cascading failures when downstream services become unhealthy.

Implement monitoring of downstream services to track latency, error rates, and other health metrics, automatically adjusting upstream behavior in response.

Track the open, closed, and half-open states for each downstream dependency. An open circuit rejects requests immediately, a closed circuit passes them through, and a half-open state allows a limited number of test requests, which can be used to determine if the service has recovered.

Provide fallback mechanisms that degrade functionality gracefully when dependencies are unavailable, rather than failing completely.

Finally, test service recovery in an automated way so you know that your system can actually recover when a problem is fixed.

Scalability

As APIs grow in popularity, scalability becomes critical. Three dimensions of scaling need attention: distributing load across existing servers, scaling horizontally by adding more servers, and auto-scaling in response to demand.

Load distribution

Choose load balancing algorithms appropriate to your workload. Common options include round-robin for even distribution, least-connections for varying request durations, and weighted algorithms for heterogeneous server capabilities.

Implement both active health monitoring (explicitly checking server health) and passive monitoring (observing request failures) to detect when servers become unhealthy.

Handle stateful operations carefully. Some applications require session persistence so that requests from a single client are always routed to the same server.

Route requests to the nearest available servers to minimize latency for geographically distributed users.

Horizontal scaling

Designing for horizontal scaling starts with implementing stateless API designs wherever possible. If all state is external (in databases or caches), then scaling is simply a matter of adding more servers.

For databases, begin with read replicas and connection pooling before attempting more complex sharding strategies.

Isolate resources between services to prevent resource contention. A resource leak in one service shouldn’t affect others.

Auto-scaling

Implement metrics-based auto-scaling that triggers when CPU, memory, or request metrics cross configured thresholds. More sophisticated approaches use predictive or proactive scaling based on historical patterns to anticipate demand spikes before they occur.

Ensure that scaling is gradual and graceful to avoid service disruption during scaling events.

Throughout all scaling decisions, balance the performance improvements gained against the resource costs incurred.

Security and privacy

Security must be built into network APIs from the start, addressing both authentication and data protection.

Authentication and authorization

Choose authentication and authorization mechanisms appropriate to your transport protocol.

For stateless HTTP APIs, token-based authentication using JWT is RECOMMENDED. Use fine-grained permission models based on scopes rather than broad roles. Scopes allow precise control over what clients can do. Set appropriate token lifetimes with expiration dates so that compromised tokens have limited utility. Implement refresh mechanisms that allow clients to renew their tokens using a secure process, typically involving a separate refresh token that can only be used to obtain new access tokens.

Data protection

Encrypt all data in transit using TLS 1.2 or higher for all communications – no exceptions!

Validate and sanitize all input parameters to prevent clients from submitting malicious data.

Prevent injection attacks through proper output encoding, so data cannot be interpreted as code.

Log security-relevant events and access patterns to create an audit trail that can help detect compromises and troubleshoot security incidents.

Logging and monitoring (observability)

Effective observability requires collecting the right metrics, logging in a structured way, and alerting on problems before they become critical.

Metrics and monitoring

Collect the "golden signals" of latency, traffic, errors, and saturation to get a holistic view of API health.

Beyond these foundational metrics, collect business metrics such as API usage, feature adoption, and user behavior to understand how the API is actually being used.

Monitor server resources and network performance as needed to understand how your infrastructure requirements are changing over time.

Implement custom, domain-specific performance indicators that are meaningful for your business.

Logging standards

Logs must be highly structured to enable automated analysis and alerting. JSON is the recommended format. Use a documented JSON schema and validate log output against it to ensure consistency.

Use correlation IDs to track requests as they flow across service boundaries, making it possible to reconstruct complete request flows across multiple services.

Implement appropriate log levels – debug, info, warning, error, and critical – and adjust verbosity controls on an environment-by-environment basis. Production logs should contain only essential information (and no PII) while development logs can be more verbose.

Define and enforce log storage and rotation policies to manage disk space and to comply with data retention requirements.

Alerting

Effective alerting catches problems quickly before they impact users.

Alert on SLA violations and performance degradation so that you know when the API is not meeting its commitments. Implement automated anomaly detection to catch unexpected patterns that might indicate a problem.

Define clear incident response procedures (also called escalation policies) so that when an alert fires, the right person is notified and knows what to do. Write runbooks to define your response plan.

Provide real-time visibility into API health through GUI dashboards.

Documentation and versioning

Well-documented APIs with clear versioning practices make it easier for clients to adopt and for you to evolve the API over time.

API documentation

Provide interactive, executable API specifications (eg. OpenAPI) that clients can use to understand your API and generate client libraries.

Consider providing client code examples in multiple programming languages to help developers get started quickly.

Maintain API changelogs that document all changes, and provide migration guides to help clients understand what they need to change when you release new versions.

Maintain an error catalog that documents all possible error conditions and responses so clients know how to handle every failure mode.

Version management

Use a conventional versioning scheme such as semantic versioning (<major>.<minor>.<patch>) so that clients can understand the impact of upgrading.

Be strict about maintaining backward compatibility within major versions. Avoid breaking changes that force all clients to update.

Publish a deprecation policy with clear timelines for deprecating features, giving clients adequate time to migrate.

Offer guidelines and tools to support migrations between major versions, making it easier for clients to upgrade when the time comes.

Testing

Comprehensive testing at multiple levels ensures that network APIs work correctly and continue to work as they evolve.

Test coverage

Implement testing at multiple levels: unit tests for discrete pieces of business logic, integration tests for service interactions and data flow, and system tests that verify individual API endpoints end-to-end.

Load testing validates that performance meets expectations under expected traffic loads.

Security testing checks for common vulnerabilities and attack vectors that could compromise your API.

Environment management

Maintain production-like testing environments (called "staging") so that you can test your API under realistic conditions before deploying.

Use representative dummy data while protecting privacy. Avoid using real customer data in test environments.

Test your deployment and rollback procedures so you know they work and can recover quickly if a deployment goes wrong.

Test system resilience under failure conditions using chaos engineering techniques to ensure your API can withstand and recover from various failure scenarios.

Compliance and governance

APIs that handle sensitive data or serve important business functions must comply with regulations and operate under clear governance.

Data governance

Regularly review your compliance with privacy regulations such as GDPR and CCPA. Define and enforce data retention policies so that you don’t keep data longer than necessary.

Classify data by sensitivity and apply appropriate controls. Highly sensitive data needs more protection than public data.

Use access controls to implement the principle of least privilege, giving each service only the permissions it needs to do its job.

API governance

Establish API design review processes so that new APIs meet your standards before they’re released. Implement automated enforcement of standards and policies to catch violations during development.

Define API lifecycle stages and gates so that APIs progress through development, testing, and deprecation in an orderly way.

Monitor API adoption and usage patterns to understand which APIs are important, identify unused APIs that can be deprecated, and spot emerging patterns in how your APIs are being used.