TS-10: Releasing

This technical standard covers best practices for releasing software updates. The focus of these guidelines is on release strategies that minimize the risk of bugs, regressions, and incidents occurring in production.

These guidelines are applicable to both software-as-a-product, which tends to be intermittently released, and software-as-a-service, which tends to be more continuously released.

This technical standard is concerned with production releases, rather than deployments to other environments such as test and staging environments.

See also TS-9: Version Control for the branching, integration, and release-trunk/release-branch mechanics that underpin these release strategies, and TS-11: Versioning for version numbering.

Release cadence

The first and most consequential decision in any release process is the cadence – how frequently new versions are delivered to users, and how much change each release contains. Cadence is not merely an operational matter – it shapes which release strategies are viable, what level of automation is required, the testing investment needed, and how the team responds to incidents.

There is a spectrum of release cadences, from rare and bundled at one end to continuous and automatic at the other. Most organizations adopt a cadence that fits their software type, their consumers' tolerance for change, and their regulatory environment. Within a single organization, different products may adopt different cadences.

Big bang

A big bang cadence delivers infrequent releases that bundle a large volume of accumulated change. Versions are typically planned and scheduled well in advance, often with formal release windows and stakeholder sign-off. Each release is a discrete, announced event.

This cadence is common for shrink-wrapped products distributed to many independent installations, for software with strong backward-compatibility commitments, and for organizations operating under strict change governance.

The principal disadvantage of a big bang cadence is the difficulty of isolating the cause of any post-release problem. When many changes ship together, attribution is hard and the blast radius is large. To compensate, big bang cadences typically rely on extensive pre-release testing, formal release rehearsals, and detailed rollback plans.

Release trains

A release train cadence delivers versions at a regular, predictable interval – such as every two weeks, every month, or every quarter. Changes that are merged and verified before the cut-off board the train. Changes that miss it wait for the next one.

Release trains offer a balance between predictability and throughput. They give consumers a known upgrade rhythm, allow downstream integrators to plan ahead, and reduce the all-or-nothing pressure of big-bang releases. The trade-off is that a feature missing its cut-off may be delayed by a full cycle.

This cadence is widely used for libraries, frameworks, mobile applications, and platforms with external consumers who need predictable timing.

Continuous deployment

In continuous deployment, every change that passes automated verification is released to production automatically, often within minutes of being merged. There is no scheduled release event – release happens as a side effect of the normal development workflow.

Continuous deployment requires high-confidence automated testing, robust observability, and the operational maturity to detect and respond to incidents quickly. It is typically combined with feature flags and progressive rollout strategies, so that the act of deploying code is decoupled from the act of releasing a feature to users.

This cadence is well suited to web services and SaaS products, where all users run the same version and there is no installer or upgrade step on the consumer side. It is generally inappropriate for software distributed to consumer devices or installed on-premises, where update timing is outside the software vendor’s control.

Cadence and versioning

The choice of cadence has direct implications for versioning. Big-bang and release-train cadences pair naturally with semantic versioning, where each release is a discrete event with a meaningful version number that consumers can reason about. Continuous deployment, by contrast, releases too frequently for human-meaningful version numbers to remain informative; in that context, build tags, commit SHAs, or date-based versioning (CalVer) are often more useful than SemVer. See TS-11: Versioning.

Release strategies

Cadence determines when and how much is released. Release strategy determines how a new version is rolled out to the users or instances that consume it. Strategies sit on a spectrum from instantaneous, all-at-once cutovers to gradual, incremental rollouts.

The simplest possible strategy is to deploy the new version to all users or instances at once – a big bang deployment. For low-risk changes, low-traffic systems, or releases timed against a scheduled downtime window, this is appropriate and requires no special infrastructure. For systems with availability requirements, larger user populations, or higher-stakes changes, one of the strategies below SHOULD be used.

The choice of strategy is constrained by the chosen cadence:

Big bang cadences typically pair with scheduled-downtime cutovers or blue-green deployment, since the release event is rare and discrete.
Release trains often pair with canary or staged rollouts, which allow time for monitoring between increments.
Continuous deployment is usually combined with rolling or canary deployment, and almost always with feature flags, so that frequent releases do not require frequent risk acceptance.

Rolling deployments

A rolling deployment gradually replaces instances running the old version with instances running the new version. At any moment during the rollout, both versions serve traffic. The system remains available throughout, because the un-updated instances continue to handle requests while updated instances come online.

Rolling deployments work well for horizontally scaled, stateless services – particularly those running on container orchestrators (Kubernetes, ECS, Nomad) or autoscaling groups, which handle instance replacement natively.

The key parameters of a rolling deployment are:

The rollout rate – how many instances are replaced at a time. Faster rollouts complete sooner but increase the impact of a faulty release.
The health check criteria – the conditions a new instance must satisfy before the rollout proceeds to the next batch.
The failure policy – what happens if a batch fails its health checks. Typically, the rollout halts and MAY automatically roll back to the previous version.

Rolling deployments require less infrastructure than blue-green deployments, since no parallel environment is maintained. However, they offer no instant rollback. Reverting requires another rolling deployment in reverse, which takes as long as the original rollout.

Rolling deployments require that the old and new versions can run simultaneously without conflict. This has implications for database schema changes, in-flight requests, message queue consumers, and inter-service contracts. Where two versions cannot safely coexist – for example, when a schema migration is not backwards-compatible – rolling deployment is not appropriate without additional design work to bridge the two versions.

Canary testing

"Canary rollout", "canary testing", or "canarying" comes from the early 20th century practice of using canaries in coal mines to detect carbon monoxide. Miners would take caged canary birds with them underground. The bird has a lower tolerance for toxic gases than humans do, so if the birds stopped chirping – or fainted – the miners knew gas was present and they needed to evacuate the mine.

In software, canary testing is a technique to reduce the risk of introducing a new version of a software service by initially rolling out the change to only a small subset of users. Common methodologies to implement canary rollouts include routing a small percentage of traffic to the new version using a load balancer, or to deploy the new version to a single node. Canary testing can also be done for software products.

An extension of canary testing is staged rollouts, in which the percentage of users receiving the new version continues to be gradually ramped up. Between each rollout stage, the health of the new version is monitored and criteria are set for the rollout to continue.

Canarying and staged rollouts are particularly useful for services that are difficult to test in isolation, or where the potential impact of a failure is high. Both were true of the 2024 CrowdStrike incident that caused a globally-disruptive outage. CrowdStrike’s software operates at the kernel level in Windows, which is inherently difficult to test in isolation and in an automated way. And such software is inherently high risk and high impact, as it has the highest level of privileges on the system. Yet the buggy update was rolled out in a single big-bang, globally.

Blue-green deployments

Blue-green deployment is a release strategy in which two identical production environments – named "blue" and "green" – are maintained side by side. At any given time, only one environment serves live traffic. Releases are performed by deploying the new version to the idle environment, validating it, and then switching traffic over – typically by updating a load balancer or DNS record.

The chief advantage of blue-green deployments is the ability to roll back near-instantaneously. If a problem is detected after the cutover, traffic can be redirected back to the previous environment without redeploying anything.

Blue-green deployments are well-suited to stateless services. For services with persistent state – particularly databases – additional care is needed to ensure that data written to the new environment remains compatible with the old, in case of rollback. This typically requires that database schema changes are applied in two phases: first, a backwards-compatible migration is deployed; second, after the new version is fully released, any cleanup migrations are applied.

Blue-green deployment differs from canary testing in that all production traffic is shifted at once, rather than incrementally. The two strategies can be combined. A new version can be deployed to the green environment and then exposed to a small percentage of traffic via canary routing before the full cutover.

Feature flags

Feature flags (also known as feature toggles or feature switches) are conditional statements in the code that determine whether a particular feature is active. The state of each flag is typically controlled at runtime, often via a configuration service, allowing features to be enabled or disabled without redeploying the software.

Feature flags decouple the deployment of code from the release of features. A new feature can be deployed to production in a disabled state, and later enabled – for all users, for a specific cohort, or gradually – without further code changes. This is a powerful tool for risk management.

Common use cases include:

Trunk-based development: Feature flags allow incomplete features to be merged to the main branch without exposing them to end users, supporting continuous integration. See TS-9: Version Control.
A/B testing: Different cohorts of users can be exposed to different variants of a feature, to evaluate which performs better.
Kill switches: A flag can be used to quickly disable a feature in production if a problem is discovered, without requiring a rollback or redeployment.
Gradual rollouts: Similar to canary testing, but operating at the feature level rather than the deployment level. A feature can be enabled for a small percentage of users initially, and ramped up over time.

Feature flags introduce complexity. Each flag is a branch in the code, and the combinatorial explosion of flag states can be difficult to test exhaustively. For this reason, flags SHOULD be used sparingly. Best practice is for an application to have a defined maximum number of flags at any one time.

Flags SHOULD have a defined lifecycle, too. Once a feature is fully rolled out and stable, the flag and the now-dead code path SHOULD be removed (via a deployment). Long-lived flags that have outlived their purpose are a common source of technical debt and, occasionally, of incidents – as when an obsolete flag is toggled accidentally.

Release approval and governance

Some contexts require formal sign-off before a release reaches production. Common drivers include regulatory compliance (financial services, healthcare, aviation, defense), contractual obligations to customers, internal change-management policies, and externally-imposed change windows. In these contexts, an intentional manual gate is required, even where the rest of the pipeline is fully automated.

The approval gate belongs at the ready → release boundary, not earlier. Earlier gates (eg. on commits to dev, or on the promotion to ready) interfere with continuous integration and the principle that the tip of ready is always shippable. Putting the gate at the release step preserves the ability to maintain a continuously-deliverable codebase while still controlling when changes actually reach production.

Approval mechanisms

Different organizations implement release approval through different mechanisms. The choice depends on the regulatory environment, the team’s tooling, and the desired trade-off between rigor and friction. Common patterns include:

Pull-request approval — A pull request to the release branch, or a tag-promotion PR, is reviewed and approved by named approvers. The PR system provides an immediate, version-controlled audit trail.
Change-management tickets — A ticket in a separate change-management system (eg. ServiceNow, Jira Service Management) is linked to the release. Approvals are recorded in the ticket; the deployment pipeline checks ticket state before promoting.
Change-advisory-board (CAB) sign-off — A scheduled review by a board of stakeholders. Common in larger or more regulated organizations, but can introduce significant latency and is poorly suited to high-cadence releases.
Automated policy gates — Rules-based gates that check predefined conditions (eg. test coverage thresholds, security-scan results, the absence of critical open incidents) and block release automatically if violated. These complement, rather than replace, human approval where governance demands it, and are essential where release cadence is too high for human-in-the-loop review.

Approval roles

Approvals SHOULD be tied to roles rather than to specific individuals, so the workflow does not break when a single person is unavailable. A named individual MAY hold the role at any moment, but the underlying authority is the role. Common roles include the on-call engineer, the release manager, the product owner, the security or compliance officer, and (for high-impact changes) an executive sponsor.

For each role granted approval authority, the organization SHOULD document:

The scope of changes the role can approve (eg. all releases, hotfixes only, releases below a certain risk threshold).
Any required sequencing — for example, security review MUST precede production approval.
Delegation rules — who acts when the primary approver is unavailable.

Audit trail

Whatever the mechanism, the approval record MUST be auditable and traceable. At minimum, the following SHOULD be captured for each release:

The version being released, with a reference to the source-control tag or commit.
The identity of each approver, the role under which they approved, and the timestamp.
The artifacts being deployed, with references to the artifact repository (see TS-9: Version Control).
Any exceptions or deviations from the standard approval process, with justification.

The audit trail SHOULD be retained for the period required by regulatory or organizational policy.

Break-glass procedures

A formal approval process MUST NOT prevent rapid response to production incidents. Even the strictest governance regime SHOULD include a documented break-glass procedure for emergency releases — typically hotfixes for security vulnerabilities, critical bugs, or active incidents.

A break-glass procedure SHOULD:

Reduce the approval requirement to a minimum (eg. a single on-call engineer, or a two-person review including the on-call engineer and a senior reviewer).
Require post-hoc review and full audit-trail backfill once the incident is resolved.
Be tested periodically, like other disaster-recovery procedures, so the team is fluent in invoking it under pressure.

The existence of a break-glass procedure is not a license to bypass normal governance — invocations SHOULD be tracked, and frequent invocations SHOULD prompt review of whether the standard process is too restrictive.

Rollback

Even with the best release practices, regressions and incidents will occur. A rollback strategy is the plan for reverting a release when problems are detected in production.

The simplest form of rollback is to redeploy the previous version of the software. This requires that the previous version’s artifacts remain available, and that the deployment process is fast enough to be useful during an incident. See TS-9: Version Control for guidance on artifact storage and the version-tag-to-artifact binding that makes rollback reproducible.

For services with persistent state, rollback is more complex. Database schema changes, in particular, are often difficult to reverse. To support rollback:

Schema migrations SHOULD be backwards-compatible across at least one release. For example, if a column is being renamed, one release can add the new column and dual-write to both, the next release can switch readers to the new column, and a later release can remove the old column.
Data migrations SHOULD be designed to be idempotent, so that they can be safely re-run if interrupted or if a partial rollback occurs.

In some cases, a roll-forward strategy is preferable to rollback. Rather than reverting to the previous version, a new release is prepared that fixes the problem. This is often the case when the rollback itself would be risky – for instance, when database state has diverged – or when a targeted hotfix can be prepared quickly.

Whatever strategy is used, the rollback procedure SHOULD be documented and periodically rehearsed. An untested rollback plan is not a rollback plan.

Change freezes

A change freeze (also "code freeze" or "release freeze") is a defined period during which no non-critical changes are released to production. Freezes are commonly applied around:

High-traffic commercial events, such as Black Friday for e-commerce platforms.
Holiday periods, when on-call coverage is reduced.
Major customer events, product launches, or marketing campaigns.
External audit or compliance windows.

During a freeze, only critical bug fixes and security patches are released. The aim is to reduce the risk of incidents at times when the cost of an outage would be unusually high, or when the team’s capacity to respond would be unusually low.

Freezes SHOULD be communicated in advance, with clear criteria for what constitutes a "critical" change and a documented exception process for unforeseen circumstances. The dates and scope of any freeze SHOULD be visible to all teams whose work may be affected.

Release documentation

Two related but distinct artifacts document what has changed in each release: release notes and changelogs. They serve different audiences and purposes, and a given project may produce one, the other, or both.

Release notes

Every production release SHOULD be accompanied by release notes that summarize what has changed. The audience and format of release notes will vary depending on the type of software:

For libraries and APIs, release notes are aimed at the developers who consume the software. They SHOULD highlight new features, deprecations, and notable bug fixes, with sufficient detail for consumers to understand the impact on their own systems. Breaking changes MUST be called out prominently and SHOULD be accompanied by a migration guide where possible.
For end-user applications, release notes are aimed at users. They SHOULD describe new functionality and notable fixes in terms that are meaningful to the user, avoiding implementation detail.
For internal services, release notes may be aimed at other teams within the organization, and SHOULD note any changes to interfaces, performance characteristics, or operational requirements.

Release notes SHOULD be written as part of the normal development process, not separately at the time of release. It is RECOMMENDED to use a consistent commit message convention to support the automatic generation of release notes, though this will not be appropriate for all software projects.

Release notes SHOULD reference the version number of the release. See TS-11: Versioning.

For projects hosted on GitHub, publishing a GitHub Release alongside the version tag is RECOMMENDED only where GitHub is the primary distribution channel — for example, for binaries, CLIs, or other artifacts that consumers download directly from the repository’s "releases" page. Where artifacts are instead distributed via a package registry (npm, PyPI, Maven Central, a container registry, etc.) or deployed directly to servers, a GitHub Release is not required. In this case, tagging the release point in Git history (see TS-9: Version Control) is sufficient, and release notes MAY be published through whatever channel reaches the actual consumers (registry changelog, internal wiki, etc.).

Changelogs

A changelog is a chronological, cumulative record of all notable changes made to a software project across its lifetime. Whereas release notes are a curated summary written for a specific audience and tied to a single release, a changelog is a comprehensive engineering reference that spans every release of the project.

The two artifacts serve different purposes and SHOULD NOT be conflated:

Release notes answer the question, "What does this release mean for me?" They are selective, narrative, and often tailored for non-technical audiences. Typical consumers include end-users, product managers, customer success teams, and – for libraries – the developers who depend on the software. Release notes for end-user products may emphasize benefits and outcomes over implementation detail, and are sometimes treated as a marketing artifact.
Changelogs answer the question, "What changed between version X and version Y?" They are exhaustive, structured, and aimed at engineers who need to reason about upgrades, debug regressions, or audit history. A changelog is a single document that grows over time, with each release adding a new section to the top.

For some projects – particularly developer-facing libraries with primarily technical audiences – the changelog and the release notes may be the same artifact. For others – particularly end-user products and large platforms – they are distinct, and the release notes are typically derived by summarizing the relevant section of the changelog in audience-appropriate language.

Use cases for each:

Maintain a changelog for any software where consumers may need to compare versions, plan upgrades, or trace when a particular change was introduced. This includes libraries, APIs, CLIs, infrastructure components, and most internal services.
Publish release notes for any software where a release is a discrete event communicated to an audience – including end-user applications, paid products, and major platform upgrades. Release notes are also appropriate for libraries and APIs that have a wide consumer base, where the changelog alone would be too dense to communicate the significance of a release.

A widely adopted convention for changelogs is Keep a Changelog, which prescribes a standard structure organized by version and grouping changes under fixed headings: Added, Changed, Deprecated, Removed, Fixed, and Security. This structure makes it easy for consumers to scan for the changes that affect them. This technical standard RECOMMENDS the Keep a Changelog convention as the default format for changelogs. A CHANGELOG.md template following this convention is available in the template repository.

These category headings are OPTIONAL. Where a release contains only a handful of changes, grouping them under headings adds ceremony without adding clarity, and a flat, unheaded list of entries is RECOMMENDED instead. Headings earn their place once a release has enough entries, spanning enough categories, that scanning the flat list becomes harder than scanning the grouped one.

Above all, a changelog SHOULD be written for humans, not machines. Its purpose is to let a reader understand, at a glance, what has changed and why it matters to them. A changelog that only reproduces data that can be mechanically extracted from git log output does not add any meaningful value to a project (see below on generating changelogs from commit history).

In keeping with this human-first purpose, a changelog SHOULD:

Provide an entry for every released version, even where a release contains only a single change. Gaps in the record undermine its use as a reliable history.
Keep the most recent version at the top, with each new release adding a section above the last. The changelog reads newest-first.
Display the release date of each version, alongside its version number – for example, ## [1.2.0] - 2026-08-10.
Make versions and section headings linkable, so that a specific release or category of change can be referenced directly – for example, from an issue, a pull request, or a support conversation.
Maintain an [Unreleased] section at the top of the file, above the most recent tagged version, capturing changes that have landed on the development trunk but not yet shipped in a release. This section is emptied into a new version heading at release time (see TS-9: Version Control).
State whether the project follows Semantic Versioning, so that consumers know what to expect from a version bump. See TS-11: Versioning.

A changelog file (typically CHANGELOG.md) SHOULD be maintained in the source repository, with entries added as part of the same change that introduces them. This keeps the changelog accurate and current, and avoids the common failure mode of attempting to reconstruct history at release time.

Where commit messages follow a consistent convention, the changelog MAY be generated automatically from the commit history. However, human curation is often still needed to produce a useful narrative. This technical standard RECOMMENDS that changelogs are at least moderated by humans, if not entirely written by hand. A changelog that only reproduces data that can be mechanically extracted from git log output does not add any meaningful value to a project.

Migration guides

A migration guide provides step-by-step instructions for consumers to upgrade from one version of a software component to another. Migration guides are most relevant when a release introduces breaking changes, significant behavioral differences, or non-trivial configuration updates.

Whereas release notes describe what has changed, a migration guide describes what consumers need to do in response. The two artifacts are complementary. Release notes SHOULD link to the relevant migration guide for any breaking change.

Migration guides SHOULD:

Be versioned to specific version transitions (eg. "Migrating from v1 to v2"). A separate guide MAY be provided for each major version transition.
Enumerate every breaking change, with before-and-after code or configuration examples where applicable.
Highlight any required ordering of steps, particularly where data migrations or downtime are involved.
Note any tooling – such as codemods or migration scripts – that can automate parts of the upgrade.

For libraries with a large consumer base, the cost of a poorly documented migration is borne many times over. Investment in a clear migration guide pays back proportionally.

Deprecation notices

A deprecation notice is a formal, forward-looking communication that a feature, API, endpoint, configuration option, or behavior will be removed in a future release. Deprecation is the mechanism by which breaking changes can be introduced gradually, giving consumers time to adapt.

Whereas release notes and changelogs are backward-looking – describing what has already happened – deprecation notices set expectations about future change. They SHOULD be issued well in advance of the planned removal.

A deprecation notice SHOULD include:

What is being deprecated, in unambiguous terms.
The release in which the deprecation takes effect.
The release in which the feature is expected to be removed, or the criteria that will trigger removal.
The recommended replacement, if any, and a reference to a migration guide.
The reason for deprecation, where useful context for consumers.

Projects SHOULD adopt and publish a deprecation policy that defines the minimum notice period. A common convention is that a feature deprecated in version vN.x will not be removed before version vN+1.0 (for major-versioned software) or before a defined number of minor releases have elapsed.

Where the language or runtime supports it, deprecation SHOULD also be signaled in code – via compiler warnings, runtime warnings, deprecation annotations, or response headers – so that consumers are alerted in their normal workflow, not only by reading documentation.

Security advisories

A security advisory is a formal disclosure of a security vulnerability that has been identified and addressed in a release. Advisories are distinct from regular release notes because they have legal, compliance, and incident-response implications, and because they are consumed by audiences – security teams, automated scanners, downstream maintainers – who may not otherwise track releases.

A security advisory SHOULD include:

A unique identifier, ideally a CVE number for vulnerabilities of public interest, or a project-specific identifier (eg. GHSA on GitHub).
A severity rating, typically using the CVSS scoring system.
A description of the vulnerability, including the affected versions and the conditions under which it can be exploited.
The fixed version(s) and any available mitigations or workarounds for consumers who cannot upgrade immediately.
Acknowledgment of the reporter, where appropriate.

Security advisories SHOULD be published through channels that downstream consumers and automated tools can reliably discover, such as the GitHub Advisory Database, language-ecosystem advisory databases (RustSec, npm, PyPI), or a dedicated mailing list.

The disclosure of a security issue SHOULD be coordinated with the release of the fix. Premature disclosure exposes users to attack; delayed disclosure prevents users from understanding why an upgrade is urgent. A documented security disclosure policy – including how to report vulnerabilities and the project’s expected response times – SHOULD be published in the repository, conventionally as SECURITY.md.