TS-49: Cloud platform engineering

This technical standard sets out some broad principles and best practices for cloud platform engineering. The guidance here is not specific to any particular cloud service provider, but is intended to be applicable to any cloud platform.

Platform engineering is a subset of cloud engineering. The focus here is on the building of self-service platforms that enable development teams to deliver applications and services more efficiently. Platform engineering is about creating a standard set of tools, processes, and best practices within an organization to enable development teams to build, test, deploy, and manage their applications themselves, with minimal ops support.

Specific technologies that support cloud platform engineering, such as infrastructure-as-code, CI/CD pipelines, and monitoring and observability tools, are covered in other technical standards. For guidance on cloud cost management and the tradeoffs between dedicated servers and cloud-native architectures, see TS-50: Cloud economics.

Guiding principles

The overarching goal of platform engineering is to accelerate delivery without adding to the workload of development teams. A well-designed platform achieves this by combining two qualities that are often treated as being in tension:

  • High autonomy
  • High alignment

Aligned autonomy

Teams need autonomy to make their own technology decisions, close to the products they are building. But autonomy alone is not enough. Those decisions also need to be aligned with the wider interests of the organization, not just the immediate needs of a single team.

These two goals are not a trade-off. The aim is not to find a comfortable midpoint between autonomy and alignment, sacrificing some of each. The aim is to maximize both. This is the principle of aligned autonomy, which means teams are free to make their own decisions, but always within well-defined global constraints, and always working in the same direction of travel.

Crucially, alignment is expressed as context, not command-and-control. Teams are given the context they need to make good technical decisions, and are then trusted to make them.

Management capacity must scale with the system

A platform exists because human management capacity does not scale on its own. One person, or one team, can reliably manage only a bounded number of systems, services, and configuration changes. As the number of objects under management grows — accounts, clusters, deployments, environments — the capacity required to manage them grows with it, and beyond a certain point no amount of additional headcount closes that gap.

Automation is the only mechanism that scales past this point. It is not an optional efficiency gain layered on top of platform engineering; it is the precondition for the organization being able to manage what it has built at all. A platform’s job is to industrialize this automation, not merely to script individual tasks.

For the automation itself to hold up under growth, the platform’s own management and CI/CD tooling MUST meet three requirements:

  1. Scalable. The tooling’s own capacity MUST grow with the number of objects it manages, or it becomes the next bottleneck.
  2. Reliable. The tooling MUST operate without introducing errors of its own — a management layer that fails unpredictably adds risk rather than removing it.
  3. Parallel. The tooling MUST be able to act on multiple objects concurrently, rather than working through a queue of changes one at a time.

Enough human resources MUST still be available alongside the automation. Automation raises the ceiling on what a fixed team can manage, but it does not remove the need for people to design, operate, and evolve the platform itself.

Two failure modes

When autonomy and alignment fall out of balance, organizations tend toward one of two failure modes.

Technology autocracy is the result of too much alignment and too little autonomy. The organization imposes rigid, iron-clad rules on technology, and a small number of technical leaders must approve every decision, no matter how small. Teams have little or no freedom to make their own design choices. Platform engineering can be misused to enforce this kind of autocracy — centrally controlling languages, application frameworks, build tools, third-party dependencies, and so on. The result is slow delivery.

Technology anarchy is the opposite – too much autonomy and too little alignment. The platform grants teams a great deal of freedom, but there is little technical alignment across the organization. Anarchy is often an overcorrection — a reaction to a previous experience of autocracy, where the instinct is to swing to the opposite extreme. But the correct response to autocracy is not to abandon alignment – it is to restore autonomy while keeping alignment intact.

The cost of anarchy is the loss of economies of scale. A platform that tries to be everything to everyone — supporting too many technology stacks — ends up maintaining multiple deployment pipelines, multiple testing environments, multiple observability dashboards and alerting configurations, and so on.

The burden of this fragmentation surfaces most painfully during large, cross-cutting changes. When migrating to a new cloud platform, for example, each team is left to manage its own migration — time spent on infrastructure work rather than on delivering planned features.

A platform therefore SHOULD NOT attempt to support an unbounded range of technology choices. Some decisions MUST be made centrally, deliberately taking that responsibility away from individual teams. This is not a constraint on autonomy so much as a precondition for it: it frees teams to spend their time on the product, rather than on maintaining bespoke delivery infrastructure.

Paved roads

The way to resolve technology anarchy is to build technical alignment into paved roads — an opinionated platform that makes the well-aligned path the easy path. Teams remain free to make their own decisions, but the platform encodes the organization’s accumulated context so that the default choices are also the right ones.

To establish this contextual alignment, ask technology leaders for three things:

  1. Their guidance on technology stack and architecture.
  2. Their expectations about how that guidance will be followed.
  3. The business consequences they anticipate — for the organization as a whole — if those expectations are not met.

This guidance SHOULD be captured in decision records. Platform capabilities are then built on the foundation of those decision records, baking the technical alignment directly into the platform. Teams are trusted to use that contextual alignment to make their own decisions — independently, but always in service of the same goals.

Multi-product deployments

A common pattern for organizations is to support independent operations of multiple products and services.

The RECOMMENDED approach is to use multiple accounts with your cloud service provider, one for each discrete product. In AWS, the AWS Organizations feature allows multiple independent AWS accounts to be managed under one root organization.

Root Organization
├── Management Account
├── Shared Services Account (networking, DNS, logging)
├── Product A - Test Account
├── Product A - Prod Account
├── Product B - Test Account
├── Product B - Prod Account
└── Security/Audit Account

In AWS, AWS Resource Access Manager (RAM) can be used to create shared VPCs/subnets across multiple accounts in the same organization, thus allowing for service-to-service communication. IAM roles can also be configured to allow for cross-account access.

A simpler solution is to use VPCs for isolation of products under the same account. VPC peering or transit gateways can be used for network connectivity between VPCs, while IAM policies can be used to control access to particular services.

Single AWS Account
├── Product A VPC
├── Product B VPC
├── Shared Services VPC
└── Use tags, IAM policies, and resource naming for separation

This is simpler to set up, and cross-product communication is easier to configure. But it does not scale as well, and isolation is weaker. It is also harder to track costs on a per-product basis.

More commonly, organizations opt for a hybrid approach. This is where products are isolated at the account level, and there is an additional account that manages centralized infrastructure that is shared by all the accounts.

Organization
├── Shared Infrastructure Account
│   ├── Shared VPC
│   ├── Transit Gateway
│   ├── Route53 Hosted Zones
│   └── Shared databases/caches
├── Product A Account
└── Product B Account

Development and testing environments

Development and testing environments SHOULD be zero-touch ephemeral environments. These are isolated environments that are automatically created and destroyed as needed, without any manual intervention.

It is RECOMMENDED to use infrastructure-as-code (IaC) tools to manage these environments. This allows for consistent and repeatable environment creation on-demand. Rollback is more easily automated, too.

Ephemeral environments are designed to be cost-effective, as they can be easily spun down when no longer needed, allowing organizations to only pay for the resources they are actively using. Automation can be used to automatically terminate most non-production environments outside of normal working hours.

Development and testing environments MUST be close replicas of production environments, with essentially the same underlying infrastructure and configuration. This ensures that any issues encountered in production environments are more likely to be caught early in development or testing, prior to deployment to production. The only differentiating factors should be the use of dummy data in non-production environments, and pre-production environments would typically have fewer resources than production environments, too.

Cloud service models and platform evolution

Platforms typically mature along a progression from raw infrastructure control toward increasingly abstracted, self-service operation. Understanding where an organization sits on this progression — and where it is heading — is useful context for the platform decisions covered elsewhere in this standard.

IaaS, PaaS, and container orchestration

The progression usually runs through three broad stages:

  1. Infrastructure-as-a-Service (IaaS). Teams provision and manage virtual machines directly, with configuration management tooling (eg. Puppet, Ansible, Chef) applied on top. Engineers retain shell access to running instances, and operational patterns resemble those of physical servers: changes are made directly on the machine, or by re-running configuration management against it.
  2. Platform-as-a-Service (PaaS). An orchestration layer (eg. Mesos/Marathon, Nomad, or a managed PaaS offering) takes over scheduling and lifecycle management. This stage is where the no-SSH operating model (see below) is typically introduced, and it is commonly the most disruptive transition in the sequence, because it removes a capability engineers have relied on since the IaaS stage.
  3. Container orchestration. A general-purpose orchestrator (most commonly Kubernetes) replaces the PaaS layer, offering a richer and more portable scheduling model. This stage does not, by itself, change the operating model established at the PaaS stage — it inherits the no-SSH, immutable-infrastructure expectations already in place.

The transition between these stages is rarely instantaneous. It is common for an organization to run stages side by side for an extended migration period, with a growing share of services moved onto the newer stage over time.

The no-SSH, immutable-infrastructure operating model

From the PaaS stage onward, engineers SHOULD NOT have interactive shell access to running production instances. Instead, all changes are made by replacing running instances with new ones built from an updated image or deployment artifact — the immutable infrastructure model.

This is a deliberate constraint, not an incidental side effect of the tooling. It removes an entire class of configuration drift: a running instance can no longer diverge from its declared definition through a manual ssh session, because there is no session to have. Every change is reproducible from source, auditable, and identical across every environment it is applied to.

The transition away from shell access is typically the most culturally disruptive step in the whole progression, more so than any subsequent move to a different orchestrator. Teams accustomed to diagnosing problems by logging into a box and inspecting it directly must adopt observability tooling — logs, metrics, and traces shipped off-instance — as a replacement diagnostic path before the transition, not after. Introducing the no-SSH constraint without first establishing that tooling leaves teams unable to debug production incidents.

Internal developer platforms

An internal developer platform (IDP) is a self-service layer, built and maintained by a platform team, that lets application teams deploy and operate their services without needing to understand the underlying orchestration layer directly. It typically exposes a curated set of golden paths — deploy a service, view its logs, check its health, roll back a release — through a single interface, whether a web console, a CLI, or both.

An IDP serves two purposes at once. It is the practical expression of the paved-road principle described in Paved roads: teams get a well-aligned default path without having to discover it themselves. And it is also an abstraction boundary — it decouples the interface application teams use from the orchestration technology underneath, which makes it possible to migrate between orchestrators (eg. from a PaaS to Kubernetes) with limited disruption to the teams building on top of the platform, because their day-to-day interface does not change even where the underlying implementation does.

Building an IDP is a significant investment, and it SHOULD be scoped to the golden paths that matter most, rather than attempting to expose every capability of the underlying platform. A platform that surfaces too much of the underlying orchestrator’s complexity through its self-service layer has not actually abstracted it.

Autoscaling

Autoscaling — automatically adjusting the number of running instances of a service in response to load — is an OPTIONAL but RECOMMENDED capability of a mature platform. Its purpose is to remove capacity management from the set of concerns a development team has to think about day to day: instances are added and removed automatically as demand rises and falls, without a developer needing to be involved in, or even aware of, the individual scaling events.

Autoscaling depends on the same operating model described above. It is only practical once workloads are stateless and instances are disposable — a consequence of the immutable-infrastructure model — since an autoscaler adding or removing instances must be free to do so without risking the loss of state that lives only on one particular instance.

References