TS-12: Quality Assurance

This technical standard defines the processes, practices, and culture through which software quality is assured.

Quality assurance (QA) is not a phase of development, nor is it the responsibility of a dedicated team. It is a continuous discipline that is woven into every stage of the software development lifecycle — from requirements analysis through design, implementation, testing (obviously), and deployment.

If debugging is the process of removing software bugs, then programming must be the process of putting them in.
– Edsger Dijkstra

The goal of QA is not to find defects after they have been introduced, but to prevent them from being introduced in the first place. Detection is necessary, but prevention is more effective and less expensive. The practices described in this standard — code review, quality gates, metrics, and environment management — are the mechanisms by which prevention is operationalized.

This standard focuses on the process of quality assurance: how quality is organized, what gates exist, how quality is measured, and how a culture of quality is sustained. For the specifics of testing, see TS-13: Functional Testing and TS-14: Performance Testing. For release strategies including canary testing and staged rollouts, which are also important quality control practices, see TS-10: Releasing.

Quality culture

Quality assurance begins with culture. No amount of tooling, automation, or process can compensate for a team that treats quality as someone else’s problem. In high-performing organizations, quality is a shared responsibility — every developer, tester, operator, and product owner is accountable for the quality of the software they deliver.

The most important cultural principle is that quality is built in, not inspected in. Testing and review are essential, but they are verification activities — they confirm the presence or absence of quality that was (or was not) established during design and implementation. The earlier a defect is introduced, and the later it is detected, the more expensive it is to fix. Prevention is always cheaper than detection.

This principle has a practical implication: quality practices MUST be integrated into the daily work of every team member, not deferred to a separate phase or delegated to a separate team. Writing tests is not a task that follows implementation — it is part of implementation. Reviewing code is not an overhead imposed on productive work — it is productive work. Monitoring production is not an operations concern — it is a development concern.

Shared ownership

Quality is not the responsibility of a "QA team." When a dedicated QA team exists, the natural tendency is for developers to treat quality as that team’s concern — writing code quickly and leaving defect detection to the testers. This dynamic produces more defects, longer feedback cycles, and adversarial relationships between development and QA.

The first rule of programming: it’s always your fault.
– Jeff Atwood

The RECOMMENDED model is one in which testing, review, and quality verification are embedded within the development team’s workflow. Every developer writes tests. Every developer reviews code. Every developer monitors the systems they build. Dedicated quality engineers may still exist, but their role is to coach, guide, and improve the quality process — not to be the sole gatekeepers of quality.

Prevention over detection

The most effective quality strategy is to prevent defects from being introduced, rather than finding and fixing them after the fact. Prevention practices include:

Requirements clarity — ambiguous or incomplete requirements are the single largest source of defects. Investing time in clear, testable requirements (see TS-1: Software Requirements Specification) directly reduces defect rates.
Design review — reviewing architectural and design decisions before implementation catches problems that would be orders-of-magnitude more expensive to fix in code.
Test-driven development — writing tests before implementation forces developers to think through requirements, edge cases, and interface design before writing production code. See TS-13: Functional Testing for detailed guidance.
Pair programming and mob programming — real-time collaboration catches defects at the moment they are introduced, when the context is fresh and the fix is trivial.
Coding standards and static analysis — enforcing consistent coding conventions and running automated static analysis on every change prevents entire classes of defect from entering the codebase.

Continuous improvement

Quality assurance is not a fixed process — it must evolve as the team, the codebase, and the product mature. Teams SHOULD regularly reflect on their quality practices and identify opportunities for improvement.

Retrospectives, post-incident reviews, and defect trend analysis are all mechanisms for learning from experience. When a defect reaches production, the response should not be limited to fixing the bug — it should include asking why the defect was not caught earlier and what change to the process would prevent similar defects in the future.

Best practices:

Treat every production incident as a learning opportunity. Conduct blameless post-mortems focused on process improvements, not individual fault.
Track where defects are introduced and where they are detected. A growing gap between introduction and detection signals a process problem.
Invest in developer education — training on testing techniques, security practices, and code quality pays compounding dividends.
Celebrate quality improvements, not just feature delivery. Teams that are recognized for reliability and craftsmanship will sustain those behaviors.

Definition of Done

The Definition of Done (DoD) is a shared checklist of criteria that MUST be satisfied before any work item — whether a user story, a bug fix, or a technical task — is considered complete. The DoD is a contract between the team and its stakeholders: it defines what "done" means, so that the word is never ambiguous.

Without a clear DoD, "done" becomes subjective. One developer may consider a feature done when the code compiles. Another may require passing tests. A third may expect documentation updates. These inconsistencies lead to incomplete work being declared finished, technical debt accumulating silently, and quality eroding over time.

The DoD is more important than a Definition of Ready (DoR). While a DoR describes the conditions under which work may begin, the DoD describes the conditions under which work has ended. The DoD is also more stable — it tends to remain consistent across work items, whereas the DoR often varies depending on the nature and complexity of each task.

The DoR is principally concerned with requirements readiness, and so is specified separately — see TS-1: Software Requirements Specification.

Defining the checklist

The DoD should be defined collaboratively by the team — developers, testers, product owners, and operations — and should be visible, explicit, and non-negotiable. Every item on the checklist should be objectively verifiable: a reviewer should be able to determine, without ambiguity, whether each criterion has been met.

A typical DoD might include criteria such as:

Code is written, compiles, and passes all existing tests.
New code has corresponding automated tests (unit, integration, or acceptance as appropriate).
All tests pass in the CI/CD pipeline.
Static analysis checks pass with no new warnings or violations.
Code has been peer reviewed by at least one other team member.
Documentation has been updated to reflect any user-facing changes.
The change has been deployed to a staging or pre-production environment and verified.
Any new configuration or environment variables are documented.
Security considerations have been reviewed (input validation, access control, data handling).

The DoD is not a bureaucratic formality. Each item should exist because its absence has, in the past, led to problems. If a criterion consistently adds no value, remove it. If new categories of defect emerge, add a criterion to prevent them.

Evolving the Definition of Done

The DoD SHOULD evolve as the team matures. A newly-formed team may start with a minimal DoD — code compiles, tests pass, code is reviewed — and expand it over time as the team builds confidence in its process. Attempting to impose a comprehensive DoD on a team that lacks the infrastructure or discipline to support it is counterproductive.

Conversely, a mature team with strong automation, comprehensive test suites, and established monitoring may expand its DoD to include criteria such as performance testing, accessibility verification, or deployment to production with feature flags.

Best practices:

Display the DoD prominently — on a wall, in the team’s wiki, or in the project’s contributing guidelines. It should be visible and referenced frequently, not buried in a document that nobody reads.
Review the DoD periodically — at retrospectives or at the start of each development cycle. Remove criteria that are no longer relevant, and add new ones as the team identifies gaps.
Apply the DoD consistently. Do not make exceptions for "small changes" or "urgent fixes." Inconsistent application erodes trust in the Definition of Done itself.
When a defect reaches production, check whether the DoD would have prevented it. If it would not, consider adding a criterion that would.

Code review

Code review is the practice of having one or more team members examine code changes before they are integrated into the shared codebase. It is one of the most effective quality assurance practices available — not because it catches every defect, but because it creates a culture of shared ownership, knowledge transfer, and collective standards.

All code changes, inclusive of test code, MUST be peer reviewed by at least one other team member before integration. For critical systems or significant changes, review by two or more reviewers is RECOMMENDED.

Purpose

Code review serves multiple purposes, and teams that treat it as merely a "bug-finding" exercise miss most of its value:

Defect detection — reviewers catch logic errors, edge cases, security vulnerabilities, and incorrect assumptions that the author may have overlooked. A fresh pair of eyes is remarkably effective at spotting issues that the author has become blind to.
Knowledge sharing — code review is one of the primary mechanisms by which knowledge spreads across a team. Reviewing code written by a colleague teaches the reviewer about parts of the codebase they may not have worked on. It also exposes the team to different approaches and techniques.
Standards enforcement — code review is where coding standards, architectural conventions, and design principles are upheld in practice. Automated tools can enforce syntax rules and formatting, but human review is necessary for evaluating design decisions, naming choices, and adherence to architectural patterns.
Mentoring — for less experienced developers, code review provides structured, contextual feedback on their work. It is one of the most effective forms of on-the-job learning.

Great developers are raised, not hired.
– Eduards Sizovs

As a developer you should strive to at least understand one level of abstraction deeper than you work on.
– Scott Davis

What to review

Effective code review goes beyond checking for syntax errors or style violations — those concerns should be handled by automated tooling. Human review should focus on the things that machines cannot evaluate:

Correctness — does the code do what it is supposed to do? Does it handle edge cases and error conditions?
Design — is the code well-structured? Does it follow the architecture and patterns established in the codebase? Is it appropriately modular and cohesive?
Clarity — is the code readable and understandable? Could a team member unfamiliar with this area of the codebase understand what the code does and why?
Test coverage — are the changes accompanied by appropriate tests? Do the tests verify meaningful behavior, or do they merely exercise code paths without asserting anything useful?
Security — does the change introduce any security risks? Are inputs validated? Are access controls enforced? Is sensitive data handled appropriately?
Completeness — does the change address the full scope of the requirement, including documentation updates, configuration changes, and migration scripts?

How to review

The quality of code review depends as much on the reviewer’s approach as on the code being reviewed.

Best practices for authors:

Keep changes small and focused. Large, sprawling changes are difficult to review thoroughly, and reviewers are more likely to miss issues. Aim for changes that address a single concern.
Provide context in the change description — what problem is being solved, what approach was taken, and why. Reviewers should not have to reverse-engineer the intent from the code alone.
Respond to feedback constructively. Review comments are not personal criticisms — they are contributions to the quality of the shared codebase.

Best practices for reviewers:

Review promptly. Delayed reviews block integration, slow delivery, and force the author to context-switch when they eventually receive feedback. Aim to complete reviews within one working day.
Be specific and constructive. "This is wrong" is unhelpful. "This does not handle the case where the input is null — consider adding a guard clause" is actionable.
Distinguish between blocking issues (must be fixed before integration) and suggestions (could be improved, but acceptable as-is). Use clear labeling to communicate the severity of each comment.
Focus on the important things — correctness, design, and security — rather than stylistic preferences that should be handled by automated formatters. Do not bikeshed.
Approve when the code is good enough, not when it is perfect. Every change is an increment, not a final draft.

Pair programming

Pair programming is a complementary practice in which two developers work together at a single workstation — one writing code (the "driver") and the other reviewing in real time (the "navigator"). Pair programming can be thought of as continuous, synchronous code review.

Pair programming is particularly effective for complex or high-risk changes, for onboarding new team members, and for tackling problems where the solution is unclear. It catches defects at the moment they are introduced, when the context is fresh and the cost of correction is lowest.

Pair programming and asynchronous code review are not mutually exclusive. Code that has been pair-programmed may still benefit from a lighter asynchronous review, particularly from someone outside the pair who can bring a different perspective.

Quality gates

A quality gate is a checkpoint in the software delivery process at which specific quality criteria must be met before work can proceed to the next stage. Quality gates are the mechanism by which quality policies are enforced — they turn aspirational standards into mandatory requirements.

Quality gates MUST be automated wherever possible. Manual quality gates are slow, inconsistent, and easily bypassed under deadline pressure. Automated gates are fast, consistent, and impartial — they apply the same criteria to every change, regardless of urgency or authorship.

Gate stages

Quality gates should be positioned at multiple points in the delivery pipeline, with progressively broader scope and higher cost:

Pre-commit

Pre-commit checks run on the developer’s machine before code is committed to version control. They provide the fastest possible feedback loop, catching issues in seconds rather than minutes.

Typical pre-commit checks include:

Code formatting and linting.
Local unit test execution.
Static analysis for common errors.

Pre-commit checks should be fast — if they take more than a few seconds, developers will disable them or work around them.

Check-in

Check-in gates run when code is pushed to a shared repository — typically as part of a continuous integration (CI) pipeline. They verify that the change integrates cleanly with the shared codebase and does not break existing functionality.

Typical check-in gates include:

Full unit test suite execution.
Static analysis with full rule sets.
Code compilation and build verification.
Dependency vulnerability scanning.
Test coverage analysis (to detect significant regressions in coverage).

Check-in gates MUST block integration if any check fails. A broken build should be treated as a team-level priority, and fixing it should take precedence over new work.

Integration

Integration gates run when code is merged into a trunk or release branch. These gates verify that the combined changes from multiple contributors work correctly together.

Typical integration gates include:

Full test suite execution (unit, integration, and system tests).
Performance regression checks.
Security scanning (SAST, DAST, dependency scanning).
Accessibility checks for UI changes.
Compliance verification for regulated systems.

Pre-deployment

Pre-deployment gates run after the build is complete and before it is deployed to a target environment. They verify that the artifact is ready for deployment and that the target environment is ready to receive it.

Typical pre-deployment checks include:

Smoke tests against the built artifact.
Configuration validation for the target environment.
Database migration verification.
Approval workflows for production deployments (where required by policy).

Automation and CI/CD

Quality gates are most effective when they are embedded in the CI/CD pipeline and run automatically. The pipeline should be configured so that:

Every code change triggers the appropriate gate checks automatically.
Failed checks block progression to the next stage. No exceptions.
Results are visible to the entire team — not just the author of the change.
Deployments to testing and staging environments are fully automated.
Deployments to production environments MAY include manual approval steps, but the deployment process itself MUST be automated. Manual deployment procedures are error-prone and unreproducible.

Best practices:

Design the pipeline so that the cheapest, fastest checks run first. If a formatting check fails, there is no point running the full integration test suite.
Keep gate criteria explicit and version-controlled alongside the code. Quality policies should be auditable and reviewable, just like application code.
Monitor gate execution times. Slow pipelines reduce developer productivity and create pressure to bypass checks. Invest in parallelization, caching, and infrastructure to keep gates fast.
Do not create "escape hatches" that allow changes to bypass quality gates. Once bypasses exist, they will be used — and the gates become meaningless.
Treat the CI/CD pipeline itself as production infrastructure. It should be monitored, maintained, and tested with the same rigor as the application it serves.

Testing environments

Testing environments are the infrastructure in which software is verified before — and after — it reaches production. The design and management of these environments is a critical aspect of quality assurance: tests can only be as reliable as the environments in which they run.

Environment hierarchy

Most organizations maintain a hierarchy of environments, each serving a distinct purpose:

Local development environments — individual developer workstations where code is written and tested during development. These environments run the application (or parts of it) locally, often with mocked or simplified external dependencies. Their primary purpose is to provide a fast feedback loop during implementation.
Integration environments — shared environments where changes from multiple developers are combined and tested together. Integration environments run the full application stack and are typically the target of CI/CD pipeline deployments.
Staging environments — pre-production environments that mirror the production configuration as closely as possible. Staging is the final verification step before release. Differences between staging and production — in infrastructure, data, configuration, or scale — undermine the value of staging tests.
Production environments — the live environment serving real users. Production is also a testing environment, as discussed below.

The key principle is environmental fidelity — the closer a testing environment is to production, the more reliable its test results are. Differences between environments are a primary source of "works on my machine" failures and false confidence from passing test suites.

Environment management

Testing environments MUST be treated as first-class infrastructure, not as afterthoughts or ad hoc creations.

Best practices:

Automate environment provisioning. Environments should be reproducible from configuration and scripts, not manually assembled. Infrastructure-as-code tools ensure that environments are consistent and can be recreated quickly.
Keep staging environments as close to production as possible. Same operating systems, same database engines, same network topology, same configuration management. Document any known differences between staging and production.
Provide developers with the ability to run meaningful tests locally. If local testing requires extensive manual setup or external dependencies that are unavailable, developers will skip testing or test inadequately.
Isolate test environments from one another and from production. Tests in one environment should not affect the state or behavior of another.
Manage test data deliberately. Environments need realistic data to produce meaningful test results, but test data must not include real user data unless properly anonymized.

Demo environments

Demo environments MUST have sufficient representative data to demonstrate all the capabilities of the application. This ensures that the software can be demonstrated effectively by non-technical teams such as sales and customer success — without requiring developer involvement or manual setup before each demonstration.

Demo environments should be refreshed regularly to ensure the data remains current and the environment reflects the latest version of the software.

Production testing

As systems grow in complexity, production testing becomes a necessity. No staging environment can fully replicate real-world conditions — live user behavior, fluctuating traffic patterns, real data volumes, and unpredictable third-party interactions. Production testing acknowledges this reality and verifies the system under the conditions that actually matter.

Production testing is not reckless. Modern architectures support incremental rollouts, real-time monitoring, feature flags, and rapid rollbacks, making production testing a manageable and routine practice. The key is to design production tests so that they minimize risk to real users while maximizing the signal gained.

Production testing methods include:

Smoke tests — automated checks that run immediately after deployment, before the release is made available to users. Smoke tests verify that the deployment was successful and that critical functionality is operational. They range from simple health checks ("is the service running?") to more substantive end-to-end workflows.
Synthetic transactions — automated scripts that simulate real user interactions in production, without actual user involvement. Synthetic transactions continuously execute predefined workflows — placing an order, authenticating a user, querying a report — to validate that critical business operations work correctly. They provide ongoing verification, not just a one-time check.
Parallel runs — a technique in which two implementations of the same functionality process the same requests, but only one produces the user-visible response. The results are compared to detect discrepancies. Parallel runs are particularly valuable during migrations or when replacing a legacy system with a new implementation.

For production release strategies — including canary releases, A/B testing, and staged rollouts — see TS-10: Releasing. For chaos engineering and resilience testing in production, see the test strategies section of TS-13: Functional Testing.

Production testing best practices:

Maintain dedicated test accounts and synthetic users in production, clearly separated from real user accounts. Test transactions must not affect real users or real data.
Ensure that test transactions are identifiable and excludable from business metrics and analytics. Mixing test and real data in reports undermines data integrity.
Monitor production continuously with logs, metrics, and distributed traces. Production testing is only valuable when the results are observable.
Always have a rollback strategy. Every production deployment should have a tested, automated mechanism for reverting to the previous version if problems are detected.

Quality metrics

What gets measured gets managed. Quality metrics provide objective evidence of the effectiveness of the team’s quality practices — and, crucially, of where those practices are falling short. Without metrics, quality assurance is a matter of intuition and opinion. With metrics, it becomes a discipline that can be tracked, evaluated, and improved.

Metrics must be chosen carefully. Poorly chosen metrics create perverse incentives — if a team is measured on test count alone, they will write trivial tests. If measured on defect count, they may argue about what constitutes a defect. The best metrics measure outcomes (the quality of the software) rather than outputs (the volume of quality activities).

Key metrics

Defect metrics

Defect metrics track the rate, severity, and lifecycle of defects:

Defect escape rate — the number of defects found in production relative to the total number of defects found. A rising escape rate signals that the team’s quality gates are not catching problems before release. This is one of the most important quality metrics.
Defect density — the number of defects per unit of code (typically per thousand lines or per module). Defect density identifies which parts of the codebase are most problematic and may need refactoring or more thorough testing.
Mean time to detect (MTTD) — how long a defect exists before it is discovered. A long MTTD for production defects suggests gaps in monitoring and alerting. A long MTTD for pre-production defects suggests gaps in testing.
Mean time to resolve (MTTR) — how long it takes to fix a defect once it is discovered. MTTR reflects both the complexity of the codebase and the effectiveness of the debugging and deployment process.

Test metrics

Test metrics evaluate the effectiveness and health of the test suite:

Test coverage — the percentage of code exercised by automated tests. Coverage is a useful indicator of untested areas, but it is a poor measure of test quality. High coverage with weak assertions provides false confidence. See TS-13: Functional Testing for detailed guidance on coverage.
Test pass rate — the percentage of tests that pass on each run. A consistently high pass rate is expected. More informative is the flaky test rate — the proportion of tests that intermittently pass and fail without code changes. Flaky tests erode trust in the test suite and SHOULD be fixed or removed promptly.
Build success rate — the percentage of CI builds that pass all quality gates. A low build success rate indicates either that the gates are too strict (unlikely, but possible) or that code quality at check-in is poor.

Process metrics

Process metrics evaluate the efficiency and effectiveness of the quality process itself:

Lead time for changes — the time from code commit to production deployment. Quality gates that are too slow or too numerous will increase lead time without proportional quality gains.
Cycle time for defect resolution — the time from defect report to deployed fix. This metric reflects the team’s ability to respond to quality issues.
Review turnaround time — the time from when a code review is requested to when it is completed. Slow reviews block delivery and signal a bottleneck in the quality process.

Using metrics effectively

Metrics are tools for learning and improvement — not for judgment or blame. They should be used to identify trends, detect regressions, and guide investment decisions.

Best practices:

Track metrics over time to identify trends, rather than reacting to individual data points. A single month’s defect count is noise; a six-month upward trend is a signal.
Make metrics visible to the team — on dashboards, in retrospectives, and in planning discussions. Metrics that are only seen by management are disconnected from the people who can act on them.
Use metrics to ask questions, not to draw conclusions. A rising defect escape rate prompts the question "why are more defects reaching production?" The metric does not provide the answer — investigation and analysis do.
Be cautious about tying metrics to individual performance evaluations. When metrics become targets, they cease to be good metrics. Developers who are penalized for low test coverage will write meaningless tests to inflate the number.
Review the metrics themselves periodically. If a metric is not driving useful conversations or improvement actions, stop measuring it and replace it with one that does.