TS-49: Cloud Platform Engineering

This technical standard sets out some broad principles and best practices for cloud platform engineering. The guidance here is not specific to any particular cloud service provider, but is intended to be applicable to any cloud platform.

Platform engineering is a subset of cloud engineering. The focus here is on the building of self-service platforms that enable development teams to deliver applications and services more efficiently. Platform engineering is about creating a standard set of tools, processes, and best practices within an organization to enable development teams to build, test, deploy, and manage their applications themselves, with minimal ops support.

Specific technologies that support cloud platform engineering, such as infrastructure-as-code, CI/CD pipelines, and monitoring and observability tools, are covered in other technical standards. For guidance on cloud cost management and the tradeoffs between dedicated servers and cloud-native architectures, see TS-50: Cloud Economics.

Guiding principles

The overarching goal of platform engineering is to accelerate delivery without adding to the workload of development teams. A well-designed platform achieves this by combining two qualities that are often treated as being in tension:

• High autonomy
• High alignment

Aligned autonomy

Teams need autonomy to make their own technology decisions, close to the products they are building. But autonomy alone is not enough. Those decisions also need to be aligned with the wider interests of the organization, not just the immediate needs of a single team.

These two goals are not a trade-off. The aim is not to find a comfortable midpoint between autonomy and alignment, sacrificing some of each. The aim is to maximize both. This is the principle of aligned autonomy, which means teams are free to make their own decisions, but always within well-defined global constraints, and always working in the same direction of travel.

Crucially, alignment is expressed as context, not command-and-control. Teams are given the context they need to make good technical decisions, and are then trusted to make them.

Two failure modes

When autonomy and alignment fall out of balance, organizations tend toward one of two failure modes.

Technology autocracy is the result of too much alignment and too little autonomy. The organization imposes rigid, iron-clad rules on technology, and a small number of technical leaders must approve every decision, no matter how small. Teams have little or no freedom to make their own design choices. Platform engineering can be misused to enforce this kind of autocracy — centrally controlling languages, application frameworks, build tools, third-party dependencies, and so on. The result is slow delivery.

Technology anarchy is the opposite – too much autonomy and too little alignment. The platform grants teams a great deal of freedom, but there is little technical alignment across the organization. Anarchy is often an overcorrection — a reaction to a previous experience of autocracy, where the instinct is to swing to the opposite extreme. But the correct response to autocracy is not to abandon alignment – it is to restore autonomy while keeping alignment intact.

The cost of anarchy is the loss of economies of scale. A platform that tries to be everything to everyone — supporting too many technology stacks — ends up maintaining multiple deployment pipelines, multiple testing environments, multiple observability dashboards and alerting configurations, and so on.

The burden of this fragmentation surfaces most painfully during large, cross-cutting changes. When migrating to a new cloud platform, for example, each team is left to manage its own migration — time spent on infrastructure work rather than on delivering planned features.

A platform therefore SHOULD NOT attempt to support an unbounded range of technology choices. Some decisions MUST be made centrally, deliberately taking that responsibility away from individual teams. This is not a constraint on autonomy so much as a precondition for it: it frees teams to spend their time on the product, rather than on maintaining bespoke delivery infrastructure.

Paved roads

The way to resolve technology anarchy is to build technical alignment into paved roads — an opinionated platform that makes the well-aligned path the easy path. Teams remain free to make their own decisions, but the platform encodes the organization’s accumulated context so that the default choices are also the right ones.

To establish this contextual alignment, ask technology leaders for three things:

1. Their guidance on technology stack and architecture.
2. Their expectations about how that guidance will be followed.
3. The business consequences they anticipate — for the organization as a whole — if those expectations are not met.

This guidance SHOULD be captured in decision records. Platform capabilities are then built on the foundation of those decision records, baking the technical alignment directly into the platform. Teams are trusted to use that contextual alignment to make their own decisions — independently, but always in service of the same goals.

Multi-product deployments

A common pattern for organizations is to support independent operations of multiple products and services.

The RECOMMENDED approach is to use multiple accounts with your cloud service provider, one for each discrete product. In AWS, the AWS Organizations feature allows multiple independent AWS accounts to be managed under one root organization.

Root Organization
├── Management Account
├── Shared Services Account (networking, DNS, logging)
├── Product A - Test Account
├── Product A - Prod Account
├── Product B - Test Account
├── Product B - Prod Account
└── Security/Audit Account

In AWS, AWS Resource Access Manager (RAM) can be used to create shared VPCs/subnets across multiple accounts in the same organization, thus allowing for service-to-service communication. IAM roles can also be configured to allow for cross-account access.

A simpler solution is to use VPCs for isolation of products under the same account. VPC peering or transit gateways can be used for network connectivity between VPCs, while IAM policies can be used to control access to particular services.

Single AWS Account
├── Product A VPC
├── Product B VPC
├── Shared Services VPC
└── Use tags, IAM policies, and resource naming for separation

This is simpler to set up, and cross-product communication is easier to configure. But it does not scale as well, and isolation is weaker. It is also harder to track costs on a per-product basis.

More commonly, organizations opt for a hybrid approach. This is where products are isolated at the account level, and there is an additional account that manages centralized infrastructure that is shared by all the accounts.

Organization
├── Shared Infrastructure Account
│   ├── Shared VPC
│   ├── Transit Gateway
│   ├── Route53 Hosted Zones
│   └── Shared databases/caches
├── Product A Account
└── Product B Account

Development and testing environments

Development and testing environments SHOULD be zero-touch ephemeral environments. These are isolated environments that are automatically created and destroyed as needed, without any manual intervention.

It is RECOMMENDED to use infrastructure-as-code (IaC) tools to manage these environments. This allows for consistent and repeatable environment creation on-demand. Rollback is more easily automated, too.

Ephemeral environments are designed to be cost-effective, as they can be easily spun down when no longer needed, allowing organizations to only pay for the resources they are actively using. Automation can be used to automatically terminate most non-production environments outside of normal working hours.

Development and testing environments MUST be close replicas of production environments, with essentially the same underlying infrastructure and configuration. This ensures that any issues encountered in production environments are more likely to be caught early in development or testing, prior to deployment to production. The only differentiating factors should be the use of dummy data in non-production environments, and pre-production environments would typically have fewer resources than production environments, too.

References

• Wiggins, A (2017). The Twelve-Factor App. — Factor X (Dev/Prod Parity) is directly relevant to the close-replica requirement in TS-49: Cloud Platform Engineering; Factor IX (Disposability) and Factor XII (Admin Processes) bear on platform-managed environment lifecycle and one-off administrative tasks.