TS-13: Functional Testing

This technical standard covers testing strategies to evaluate the correct operation of a software system.

A system is said to be "correct" when it performs its intended functions correctly, across various scenarios including normal operation, edge cases, and error conditions.

Functional testing is the cornerstone of software quality assurance. It involves recording the behaviors observed in user workflows and business logic against expected outcomes as specified in the system requirements.

The trouble with programmers is that you can never tell what a programmer is doing until it’s too late.

– Seymour Cray

Testing MUST be treated as a fundamental and integral part of the development process itself — not as a separate phase that follows implementation. All test code is subject to the same coding standards as application code.

Test strategies

A test strategy is a high-level approach to organizing and deploying tests to maximize their effectiveness. Where test types define what is being tested and test levels define where in the architecture testing occurs, test strategies address how and when tests are deployed across the development lifecycle.

Effective testing requires more than just writing good, individual tests. It requires thoughtful strategies for when to run tests, what to prioritize, and how to gain confidence that the tests themselves are reliable. A well-chosen combination of strategies ensures that testing effort is allocated where it will have the greatest impact — catching bugs before users do, preventing regressions, enabling confident refactoring, and building trust with stakeholders.

This section covers the most important cross-cutting test strategies.

Regression testing

Regression testing is the practice of re-running existing tests after code changes to ensure that previously working functionality has not broken. A "regression" is a defect introduced by a change that causes something that used to work to stop working. Regression testing is the primary defense against this class of defect.

Any type of test — unit, integration, system, or acceptance — can serve as a regression test. The "regression" label describes the practice of repeatedly running tests to detect unintended side effects, regardless of the test’s scope or type. A suite of unit tests run after every commit is being used for regression testing. A full integration test suite executed before each release is also regression testing.

Regression testing is most critical after:

  • Bug fixes — to confirm the fix does not break other functionality.
  • Feature additions — to verify new code does not interfere with existing behavior.
  • Refactoring — to ensure structural changes preserve external behavior.
  • Dependency updates — to catch incompatibilities introduced by third-party changes.
  • Infrastructure changes — to detect issues arising from environment modifications.

Best practices:

  • Maintain an automated regression test suite that can execute quickly and frequently. Speed is essential; slow suites get run less often, reducing their value.
  • Integrate regression tests into continuous integration pipelines so they run automatically on every code change.
  • Prioritize regression tests based on the criticality of the functionality they cover and the likelihood that it will be affected by changes.
  • Review and update the regression suite regularly. Remove obsolete tests that no longer reflect actual requirements, and add coverage for new features and recently fixed bugs.
  • Use version control and diff tools to identify which areas of the codebase have changed, and focus regression testing effort accordingly.

Smoke testing

Smoke testing, also known as build verification testing, is a strategy for quickly determining whether a new build is stable enough to warrant further testing. The term originates from hardware testing: when a new circuit board was powered on, if it literally started smoking, there was no point in running more detailed tests!

Smoke tests cover only the most critical functionality and the most common user paths. They are not thorough; their purpose is to catch catastrophic failures early, so that time is not wasted on detailed testing of a fundamentally broken build.

Smoke testing is most valuable as the first stage in a testing pipeline. If the smoke tests fail, the build is rejected immediately, and detailed testing does not proceed.

Best practices:

  • Keep smoke tests fast — they should complete within minutes, not hours.
  • Cover only the critical paths: core functionality, major user workflows, and essential integrations.
  • Automate smoke tests and run them as the first step after every build.
  • Reject any build that fails smoke testing for immediate investigation and repair.
  • Update smoke tests whenever the definition of "critical functionality" changes.

Negative testing

Negative testing — sometimes called error-handling testing or failure-path testing — is the strategy of intentionally testing a system with invalid inputs, unexpected conditions, and error scenarios. Where most tests verify that the system does what it should do, negative tests verify that the system handles gracefully what it should not have to do.

Negative testing is essential because real-world systems are constantly exposed to unexpected inputs, malformed data, network failures, and user mistakes. A system that works correctly under ideal conditions but crashes or behaves unpredictably under adverse conditions is not fit for production.

Areas of focus for negative testing include:

  • Invalid inputs — empty fields, values outside expected ranges, wrong data types, excessively long strings, special characters, and injection payloads.
  • Boundary conditions — values at the exact edges of valid ranges, where off-by-one errors and overflow conditions are most likely.
  • Resource exhaustion — out-of-memory conditions, full disks, connection pool exhaustion, and timeout scenarios.
  • Concurrency issues — race conditions, deadlocks, and data corruption under simultaneous access.
  • Dependency failures — unavailable databases, network timeouts, corrupted responses from external services.

Best practices:

  • Design negative test cases systematically. For each input, consider what happens with null values, empty values, boundary values, and malformed values.
  • Verify that error messages are informative for users and operators without exposing sensitive system internals such as stack traces, database schemas, or internal paths.
  • Ensure that errors are logged with sufficient detail for troubleshooting, including timestamps, context, and correlation identifiers.
  • Confirm that the system maintains data integrity when errors occur — partial operations should be rolled back or handled atomically.
  • Validate that error-handling paths do not introduce security vulnerabilities, such as failing open on authentication errors.

Exploratory testing

Exploratory testing is a strategy in which testers simultaneously learn about the system, design tests, and execute them — without following predetermined test scripts. It is an inherently creative and adaptive approach that leverages human intuition and curiosity to discover issues that scripted tests might miss.

Exploratory testing is particularly effective for:

  • Evaluating new or unfamiliar features, where the tester’s fresh perspective can reveal usability issues and unexpected behaviors.
  • Discovering edge cases and interaction effects that were not anticipated in the requirements.
  • Supplementing automated regression tests with the kind of unscripted, intuitive investigation that only humans can provide.

Exploratory testing is not ad hoc or random. It is most effective when conducted in structured sessions with clear objectives, time limits, and systematic documentation of findings.

Best practices:

  • Allocate specific time-boxed sessions for exploratory testing, with a defined theme or area of focus for each session.
  • Document findings immediately, including steps to reproduce any issues discovered.
  • Encourage testers to deviate from planned paths when they notice interesting or unexpected behavior.
  • Use exploratory testing as a complement to — not a replacement for — automated tests.
  • Rotate testers across different areas of the system to bring fresh perspectives.

Risk-based testing

Risk-based testing is a strategy for allocating test effort in proportion to the risk associated with different parts of the system. Not all code carries the same potential for damage if it fails, and not all code is equally likely to contain defects. Risk-based testing recognizes these asymmetries and directs attention to where it will have the greatest impact.

Risk is typically assessed along two dimensions:

  • Likelihood of failure — how likely is this component to contain defects? Factors include code complexity, frequency of changes, developer experience, and historical defect rates.
  • Impact of failure — how severe would the consequences be if this component fails? Factors include the number of users affected, financial impact, regulatory implications, and safety considerations.

Components that score high on both dimensions warrant the deepest and most rigorous testing. Components that score low on both may require only basic validation.

Best practices:

  • Assess risk early in the project lifecycle and revisit the assessment as the system evolves.
  • Use risk assessments to guide decisions about test types, test depth, and automation investment — not just test prioritization.
  • Focus the most comprehensive testing — including edge cases, boundary conditions, and negative testing — on high-risk components.
  • Accept lighter testing for low-risk, low-change components, but ensure that basic path coverage is maintained even for these.
  • Track defect data over time to validate and refine risk assessments. Areas with historically higher defect rates should be re-evaluated as higher risk.

Mutation testing

Mutation testing is a strategy for evaluating the quality and effectiveness of a test suite. The central question it answers is: how good are our tests?

The technique works by introducing small, deliberate defects — known as mutations — into the code being tested. Each mutation creates a slightly altered version of the program, called a mutant. If the existing tests detect the mutation (ie. at least one test fails), the mutant is said to be "killed." If all tests continue to pass despite the mutation, the mutant "survives" — indicating a gap in the test suite.

The mutation score is the percentage of mutants that are killed. A high mutation score indicates a test suite that is sensitive to defects and therefore more likely to catch real bugs. A low score indicates tests that may be superficial — passing regardless of correctness.

Common mutations include:

  • Replacing arithmetic operators (eg. + with -).
  • Inverting conditional expressions (eg. > with <=).
  • Removing method calls or return statements.
  • Replacing constants with different values.
  • Negating boolean expressions.

Mutation testing is computationally expensive, since each mutation requires a separate test run. For large codebases, it is typically applied selectively rather than exhaustively.

Best practices:

  • Focus mutation testing on the most critical and complex areas of the codebase, where test quality matters most.
  • Use mutation testing after bug fixes. Reverting the fix should produce surviving mutants if the test suite does not adequately cover the bug — a clear signal that more tests are needed.
  • Run mutation testing on integration of modified code, as a quality gate in the CI/CD pipeline.
  • Investigate surviving mutants. Each one represents either a genuine gap in test coverage or an equivalent mutation (a change that does not affect the program’s observable behavior). Both are informative.
  • Use mutation testing tools appropriate to your language and ecosystem. Most mainstream languages have mature mutation testing frameworks.

Chaos testing

Chaos testing — also known as chaos engineering — is a strategy for proactively testing a system’s resilience by deliberately introducing failures into a production or production-like environment. The goal is to discover weaknesses in the system’s ability to withstand and recover from turbulent conditions before those conditions occur naturally.

The foundational principle of chaos engineering, as articulated by Netflix, is that the best way to build confidence in a system’s resilience is to test it under realistic failure conditions. Hypothetical reasoning about what might go wrong is no substitute for empirical evidence of what actually happens when things do go wrong.

Chaos experiments typically involve:

  • Infrastructure failures — shutting down virtual machines, killing containers, simulating data center outages.
  • Network disruption — introducing latency, packet loss, DNS failures, or partitioning between services.
  • Resource pressure — filling disks, exhausting memory, saturating CPU.
  • Dependency failures — making external services return errors, respond slowly, or become unavailable.
  • Clock manipulation — skewing system clocks to expose time-dependent bugs.

Chaos testing differs from traditional failure testing in that it is typically conducted in production or near-production environments, under controlled conditions, with the explicit intent of learning about the system’s real-world behavior rather than just verifying expected behavior.

Best practices:

  • Start small. Begin with low-impact experiments in non-production environments before progressing to production experiments.
  • Define a steady state — the normal, healthy behavior of the system — and measure the impact of each experiment against it.
  • Automate experiments to run regularly, so resilience is tested continuously rather than as a one-off exercise.
  • Establish clear procedures for halting experiments if they cause unacceptable impact.
  • Use the results of chaos experiments to drive architectural improvements, not just incident response.
  • Build a culture in which controlled failure is seen as a learning opportunity, not a risk to be avoided.

Test types

This section defines the main types of functional testing, organized by how they approach verification of the system-under-test.

Where test strategies address how and when tests are deployed, and test levels address where in the architecture testing occurs, test types define what quality attribute is being tested and how the verification is performed. This section focuses on test types relevant to functional correctness. For non-functional test types — including performance, security, accessibility, usability, compliance, and recovery testing — see TS-14: Performance Testing.

Static analysis

Most types of tests are dynamic, which means they require the system-under-test to be compiled and executed, so the tests can make assertions on the expected dynamic behaviors of the system.

Static analysis examines the state of the code without executing it. It is possible to identify many categories of potential defects, security vulnerabilities, code quality issues, and standards compliance violations by analyzing the static structure of the source code.

Static analysis tests are easy to automate, and the automated tests are cheap and fast to run because they do not require build steps or isolated test runtime environments. For these reasons, static analysis tests are often deeply integrated into the development process, commonly run automatically when code changes are committed, checked in, and/or integrated.

Best practices:

  • Run static analysis checks on revisions (commits), check-ins (pushes to shared repositories), and integrations (before and after merges into trunks).
  • Block integrations until all static analysis checks pass.
  • Establish clear coding conventions and configure static analysis tools to enforce them consistently.
  • Track static analysis metrics over time to measure code quality improvements.
  • Use a variety of static analysis tools that specialize in different things such as coding conventions, security, and dependency analysis.

Behavioral testing

Behavioral testing, also known as black-box testing, focuses on testing the behavior of the system without considering the internal implementation details. Behavior tests validate that the system produces expected outputs for given inputs.

Behavioral testing is ideal for requirements verification. It can also be used to verify smaller components of the system, such as individual functions or modules, and integrations between those components. Thus, behavioral testing is typically undertaken at multiple levels of abstraction: unit, integration, system, and end-to-end acceptance tests. Testing levels are covered in more detail in the test levels section.

Best practices:

  • Design test cases based on requirements specifications and user stories. For unit and integration tests, the requirements are those of the components-under-test rather than the user-oriented requirements of the system itself.
  • Prefer to use dummy data that represents realistic production scenarios.
  • Aim for high path coverage. But it’s more important to cover edge cases, boundary values, invalid inputs, and error conditions.
  • Behavioral tests may step into white-box testing, where appropriate. This is particularly beneficial in lower-level tests (unit and integration) rather than higher-level tests (system and acceptance).

White-box testing

White-box testing, also known as internal testing, examines internal logic paths and data flows. White-box testing is useful for testing complex algorithms, and for checking input validation and error handling within individual components.

But white-box testing is perhaps most often used for achieving comprehensive path coverage – for ensuring that each logical branch executes at least once. Code coverage analysis tools can be used to systematically identify untested paths, and for these missing execution paths to be "filled in" with white-box tests.

White-box testing is commonly used in combination with black-box behavioral testing. It is often the case that these two types of tests will be interwoven in the same test suites, and particularly in unit tests. Together, these two types of tests can provide the highest level of confidence in the correctness of the system-under-test.

Black-box plus white-box testing form the foundation of effective testing strategies.

Best practices:

  • Strive for comprehensive path coverage – that all the critical paths are executed at least once during execution of the test suite. Tests should prioritize the critical paths plus edge cases, boundary values, invalid inputs, and error conditions. This is more important than achieving 100% coverage (which will be unattainable in many cases anyway).
  • Do combine white-box tests with black-box tests in the same tests for a single component or integration.
  • As a general rule, white-box tests are not appropriate for higher-level tests (system and acceptance).

Approval testing

Approval testing — also known as snapshot testing or golden master testing — validates that the system produces exactly the same output each time it is run for a given input. The technique works in two phases:

  1. Baseline capture — the test exercises the system-under-test and records the output as a reference result (the "approved" snapshot).
  2. Comparison runs — subsequent test runs exercise the system in exactly the same way and compare the output against the approved snapshot. If there is any difference, the test fails.

Approval testing is particularly valuable for stabilizing existing code that has little or no test coverage. When you do not fully understand what a piece of code does, you can capture its current behavior as a baseline and then use the approval tests as a safety net while refactoring. Any unintended change in behavior will be flagged immediately.

Approval testing is also well-suited to validating the visual appearance of graphical user interfaces. Specialized tools can capture screenshots of UI components and compare them pixel-by-pixel across test runs, catching unexpected visual regressions.

The limitation of approval tests is inherent in their design: they only verify that the code does what the code did before. They do not verify that the code does what it should do. For this reason, approval tests are a useful complement to — but not a replacement for — behavioral and acceptance tests.

Best practices:

  • Use approval testing as a first step when stabilizing legacy code. Capture the current behavior, then refactor with confidence.
  • Store approved snapshots in version control alongside the test code, so changes to expected output are reviewed as part of normal code review.
  • When an approval test fails, inspect the difference carefully. If the change is intentional, update the approved snapshot. If it is unintentional, investigate the regression.
  • For UI approval testing, accept that minor rendering differences (sub-pixel antialiasing, font rendering variations) may cause false failures across environments. Configure comparison thresholds appropriately.
  • Prefer behavioral tests for new development. Reserve approval testing for situations where the expected behavior is best defined by example output rather than by specification.

Test levels

Where test types define what quality attribute is being tested, test levels define where in the system’s architecture the testing occurs — from individual components in isolation, up through integrations between components, to the complete system operating as a whole.

Software systems are typically composed of many components organized in layered architectures. Testing at a single level of abstraction is not sufficient. Defects can exist within individual components, in the interactions between components, or in the emergent behavior of the complete system. Each test level addresses a different class of defect.

The test levels described here — unit, integration, system, and acceptance — form a progression from narrow, isolated tests to broad, holistic tests. As the level increases, so does the scope of the system-under-test, the number of real dependencies involved, and the fidelity of the test to production conditions. But so too does the cost: higher-level tests are typically slower to execute, harder to set up, harder to debug when they fail, and more sensitive to environmental factors.

Unit tests

Unit tests verify individual components or functions in isolation. A "unit" is typically the smallest testable part of the system — a function, a method, a class, or a module — depending on the language and the architecture.

The purpose of unit testing is to validate that each unit performs its intended function correctly. Because unit tests operate on small, isolated pieces of code, they execute very quickly and provide precise, localized feedback about where a defect has been introduced. This makes them an excellent tool during development: when a unit test fails, the source of the problem is usually obvious.

Unit tests are as much a design tool as a quality assurance tool. The practice of writing unit tests — and especially writing them first, as in test-driven development — forces developers to think about the interface and responsibilities of each component. Code that is difficult to unit test is often tightly coupled, overly complex, or poorly modularized. Testability problems tend to point to underlying design problems.

Dependencies of the unit-under-test are commonly replaced with test doubles (stubs, mocks, fakes) to ensure the test is exercising only the unit’s own logic. However, as discussed in the test doubles section, test doubles should be used judiciously. Lightweight doubles are preferred, and real dependencies should be used wherever practical to maintain high fidelity.

Best practices:

  • Each unit test should verify one specific behavior. Prefer many small, focused tests over fewer large ones.
  • Unit tests should be fast — a suite of hundreds or thousands of unit tests should complete in seconds. If unit tests are slow, they will not be run frequently enough to be useful.
  • Minimize the use of test doubles. Replace only those dependencies that are slow, non-deterministic, or unavailable in the test environment. Use real implementations wherever practical.
  • Name tests descriptively so that a failing test name alone communicates what behavior is broken.
  • Unit tests should not depend on external state such as databases, file systems, or network services. If a unit test requires external infrastructure, it is probably an integration test.
  • Organize the body of each test into three blocks: given (preconditions and setup), when (the action under test), and then (assertions about the outcome). This structure makes the intent of the test immediately clear and encourages a single action per test.

Integration tests

Integration tests verify that multiple components work together correctly when combined. Where unit tests validate components in isolation, integration tests validate the interactions, interfaces, and data flows between components.

Many defects manifest not within individual components, but at the boundaries between them — mismatched data formats, incorrect assumptions about call sequences, transaction management issues, and protocol misunderstandings. Integration tests are specifically designed to catch these boundary defects.

In practice, software systems have multi-layered architectures, so integration tests themselves exist at multiple levels of abstraction. A test that verifies the interaction between two classes is a lower-level integration test. A test that verifies the interaction between an application service and a database is a higher-level integration test. Both are integration tests, but they differ significantly in scope, speed, and setup complexity.

There are two broad approaches to integration testing:

  • Bottom-up integration — the lowest-level components are tested first, and higher-level components are added incrementally. Lower-level components are real; higher-level components that are not yet under test may be exercised through "drivers" (test harnesses that invoke the integrated components).
  • Top-down integration — the highest-level components are tested first, with their lower-level dependencies replaced by stubs. Real implementations are substituted in as lower layers are integrated.

In practice, most teams use a pragmatic combination of both approaches.

It is worth noting that integration tests tend to be more tactical than strategic. If a project has comprehensive unit tests and well-designed acceptance or system tests, the integrations between components are already being exercised — indirectly — through those higher-level tests. Integration tests are most valuable when they are added to address specific, known sources of integration failure: a flaky external dependency, a complex data transformation at a service boundary, or a historical pattern of contract-breaking changes. They provide an early warning system for these specific risks, failing faster and more precisely than a system test would.

Best practices:

  • Focus integration tests on the boundaries and interfaces between components — the data that crosses those boundaries, the contracts that govern them, and the error handling at those boundaries.
  • Use real dependencies wherever practical. The closer the test environment is to production, the more likely integration tests are to catch real defects.
  • Integration tests will be slower than unit tests. Organize them so they can be run both as part of the full test suite and selectively for specific subsystems during development.
  • Where test doubles are necessary (eg. to replace an external service that is unavailable in the test environment), prefer contract-based approaches that verify both sides of the interface independently.
  • Ensure integration tests cover failure paths — not just the happy path. What happens when a downstream service times out? When a database connection is lost? When a message is malformed?

System tests

System tests validate the entire application as a complete, integrated system. They evaluate the system’s compliance with its specified functional and non-functional requirements in an environment that resembles production as closely as possible.

System tests operate at the highest level of the application’s own architecture. They exercise the system through its external interfaces — whether that is a user interface, an API, a command-line interface, or a message queue. Unlike unit and integration tests, which test internal components and their interactions, system tests treat the application as a black box and verify its end-to-end behavior.

System tests are sometimes referred to as end-to-end tests or feature tests, although these terms are used inconsistently across the industry. In this standard, "system tests" refers to tests that validate complete workflows through the full application stack in a production-like environment.

Because system tests involve the entire stack, they tend to be the slowest and most expensive tests to run. They are also the most sensitive to environmental differences between test and production. But they provide the highest fidelity: a passing system test provides strong evidence that a complete user workflow functions correctly.

Best practices:

  • Design system tests around complete user workflows and business scenarios, not around individual components or internal structures.
  • Run system tests in environments that mirror production as closely as possible — same operating system, same database engine, same network topology.
  • Keep the number of system tests manageable. System tests are expensive; use them to verify critical paths and high-risk scenarios, and rely on unit and integration tests for breadth of coverage.
  • Automate system test execution and integrate them into the CI/CD pipeline. Manual system testing does not scale and is prone to inconsistency.
  • When system tests fail, invest in making the failure output diagnostic. System test failures can be difficult to debug because of the large scope; detailed logs, screenshots, and request traces are essential.

Acceptance tests

Acceptance tests confirm that the system-under-test meets business requirements and is ready for deployment. They answer the question: does this system do what the customer or stakeholder asked for?

Acceptance testing is distinct from system testing in its perspective and ownership. System tests are typically written and maintained by the development or QA team and verify the system against technical specifications. Acceptance tests are defined in collaboration with stakeholders and verify the system against business requirements — often expressed as user stories, acceptance criteria, or business rules.

There are two common phases of acceptance testing:

  • Alpha testing is conducted at the development site in a controlled environment. It typically involves customer representatives or product owners using the system under close supervision from the development team. Alpha testing occurs after system testing is complete and is focused on validating that the system meets its specified requirements.
  • Beta testing is conducted at the customer’s site — or by external users — in a real-world environment. The development team is not directly supervising. Beta testing exposes the system to a broader range of usage patterns, environmental conditions, and edge cases than are typically covered in alpha testing.

Best practices:

  • Define acceptance criteria collaboratively with stakeholders before implementation begins. Acceptance criteria should be specific, measurable, and testable.
  • Automate acceptance tests where possible, using tools that allow tests to be expressed in terms that stakeholders can understand (eg. behavior-driven development frameworks).
  • Conduct alpha testing with customer representatives who will actually use the system, not with developers or testers acting as proxies.
  • For beta testing, recruit a representative group of users and provide a structured mechanism for collecting and triaging feedback.
  • Establish clear criteria for acceptance — what constitutes a "pass" or "fail" — before testing begins. Ambiguous acceptance criteria lead to disputes and delays.
  • Acceptance tests should focus on what the system does, not how it does it. They should be resilient to internal refactoring.

Behavior-driven development

Behavior-driven development (BDD) is the RECOMMENDED approach to writing acceptance tests. BDD is not about using a particular tool — such as Cucumber or Behat — but about expressing tests in the language of the business domain, so that they function as executable specifications of the system’s intended behavior.

When acceptance tests are written as executable specifications, they serve a dual purpose: they verify that the system does what the stakeholders asked for, and they document what the system is supposed to do. This makes them enormously valuable. The specifications are always up-to-date, because they are verified by every test run. And because the tests are written in domain language, they can be reviewed, critiqued, and even co-authored by non-technical stakeholders.

The key discipline of BDD is to keep tests focused on the problem — the desired behavior — and not on the solution — the implementation details. A well-written BDD test should not mention button labels, form fields, URLs, or any other implementation artifact. The same executable specification should be valid whether the system is implemented as a web application, a CLI tool, or a mobile app. This separation means that the specification only changes when the understanding of the problem changes, not when the implementation is refactored.

Software development is most effective when it is focused on outcomes over implementation. BDD complements this by driving development from the perspective of the user’s goals. The first act of starting work on a new feature should be to identify one or more examples that demonstrate the feature in action, expressed as executable specifications. This should be treated as a normal, everyday part of the development process.

Acceptance tests are also the easiest kind of test to retrofit to existing code. Because they are black-box tests that interact with the system only through its public interfaces, they do not require access to internals. This makes them a valuable tool for stabilizing legacy systems before beginning refactoring.

Best practices:

  • Write acceptance criteria in domain language, collaboratively with stakeholders, before implementation begins.
  • Do not couple acceptance tests to specific user interfaces or implementation details. The tests should not reference forms, buttons, URLs, or database tables.
  • Use BDD as a forcing function for incremental development — write the specification, implement just enough to satisfy it, and repeat.
  • Treat executable specifications as the authoritative definition of the system’s intended behavior. When a specification and the system disagree, the specification is right and the system is wrong.

The test pyramid

The test pyramid is a widely-referenced model for thinking about the relative distribution of tests across levels. In its traditional form, the pyramid has a broad base of unit tests, a narrower middle layer of integration tests, and a small peak of system and acceptance tests. The idea is that most defects should be caught by fast, cheap unit tests, with fewer — but broader — tests at higher levels to catch integration and end-to-end issues.

The test pyramid is a useful heuristic, but it should be understood as a descriptive observation rather than a prescriptive target. The shape of the pyramid is the natural outcome of a bottom-up design process: when a system is built component by component, and each component is test-driven, the result is many unit tests and relatively few system tests.

A top-down approach — in which development begins with high-level (failing) system or acceptance tests and works downward — tends to produce a different distribution. In this model, more coverage is achieved through end-to-end tests, and unit tests are written only for the most complex or brittle components. The result is fewer unit tests to maintain, which can make large-scale refactoring easier.

Neither distribution is inherently superior. The right balance depends on the architecture of the system, the development methodology, and the risk profile of the project. What matters is not the shape of the pyramid, but that every level provides meaningful signal — catching the defects for which it is best suited — without excessive redundancy between levels.

Regardless of the overall distribution, unit tests and acceptance tests are the two most important testing levels and should be at the center of any testing strategy. Unit tests drive the design of individual components and catch fine-grained defects quickly. Acceptance tests validate that the system achieves its intended outcomes from the user’s perspective. When development is driven from these two levels — acceptance tests that define what the system should do, and unit tests that drive how it is built — the result is a robust, well-specified system. Integration and system tests are valuable complements, but they are best used tactically to address specific risks rather than as the primary mechanism for quality assurance.

Test coverage

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence […​]. If I don’t typically make a kind of mistake (like setting the wrong variables in a constructor), I don’t test for it.

– Kent Beck

Test coverage — also known as code coverage — is a measure of how much of the codebase is executed by the test suite. Coverage metrics help identify untested areas of the code, but they must be interpreted carefully: high coverage does not guarantee good tests, and low coverage does not necessarily indicate poor quality tests either.

Coverage can be measured at several granularities:

  • Statement coverage measures whether each statement in the code has been executed at least once. This is the most basic and most commonly reported coverage metric.
  • Branch coverage (also called condition coverage or decision coverage) measures whether each branch of every conditional expression has been evaluated as both true and false. Branch coverage is strictly more rigorous than statement coverage — it is possible to achieve 100% statement coverage while missing branches entirely.
  • Path coverage measures whether every possible execution path through the code has been traversed. Path coverage is the most thorough metric, but it is often impractical to achieve fully, because the number of possible paths grows exponentially with the number of branches.

Coverage across test levels

An important principle is that coverage is not the exclusive concern of unit tests. Coverage can — and should — be achieved through tests at multiple levels.

A system test that exercises a complete user workflow will execute code paths across many components, contributing to statement and branch coverage just as a unit test would. Similarly, integration tests contribute coverage at the boundaries between components. The coverage tool does not distinguish which level of test exercised a given line — it simply records that the line was executed.

This means that a project does not need to unit test every function to achieve high coverage. If a particular code path is already well-covered by integration or system tests, adding a redundant unit test for the same path adds maintenance cost without proportional benefit. Conversely, complex branching logic deep inside a component may be impractical to cover through high-level tests alone, and is best covered by targeted unit tests.

The goal is to achieve adequate coverage through the most appropriate combination of test levels — not to maximize coverage at any single level.

Coverage targets

Coverage targets — such as "80% statement coverage" — are a common practice but should be applied thoughtfully. A rigid coverage target can incentivize the wrong behavior: writing low-value tests that exercise trivial code simply to satisfy the metric, while leaving genuinely complex or risky code undertested.

Treat coverage as a diagnostic tool, not a goal in itself. A coverage report is most useful for identifying gaps — areas of the codebase that are not exercised by any test — so those gaps can be evaluated and addressed where appropriate. Not every gap needs to be filled; some code is genuinely trivial and does not warrant a dedicated test.

Best practices:

  • Use coverage reports to identify untested code, then make a deliberate decision about whether that code warrants testing. Focus on complex logic, error-handling paths, and high-risk components.
  • Measure branch coverage in addition to statement coverage. Statement coverage alone can give a misleading picture of test thoroughness.
  • Do not pursue 100% coverage as an end in itself. The effort required to cover the last few percent almost always exceeds the value gained.
  • Track coverage trends over time. A declining coverage trend — especially in areas of active development — is a more useful signal than any absolute number.
  • When setting coverage thresholds, apply them per-component or per-module rather than as a single project-wide number. Critical modules may warrant higher thresholds than utility code.

Test doubles

A test double is a generic term for any object that stands in for a real object in a test. The real objects, which are swapped out for doubles in test scenarios, are dependencies of the component-under-test.

The purpose of swapping out real dependencies with doubles is to be able to isolate the behavior of the component-under-test from its dependencies. This allows tests to verify specific units of behavior, rather than testing the integrated behaviors of multiple components.

The lower the level of the test, the more likely test doubles will be useful. Therefore, unit tests tend to have more doubles than integration tests, and system tests may have few or none at all.

Types of doubles

There are several ways that test doubles can be implemented. The following terms refer to different types of test doubles:

  • Mocks
  • Stubs
  • Dummies
  • Fakes
  • Spies

There are no definitive definitions of these terms. Test utilities tend to use them interchangeably. But here are the common-agreed definitions:

A fake is one of the most advanced types of test double. Fakes will often be fully functioning implementations of the interfaces of the components they replace. They tend to be developed and maintained alongside the real components in the application code, whereas most other types of doubles tend to be defined in the test code. Fake implementations will take some shortcuts in their implementation, so they can be run efficiently and not depend on external systems, which may not be available in test environments. For example, the fake implementation of a database repository may implement an in-memory store, instead. But otherwise fakes will replicate the behavior of the real implementations as closely as possible.

A stub can be thought of as a lightweight fake. Like a fake, it implements the interface of the component it is replacing. But all it does is provide canned answers to calls made during the test — nothing more elaborate than that. Unlike a mock, a stub does not make any assertions itself. And because its responses are hard-coded, the test code will not make any assertions on the stub’s behavior, either. Stubs are the lightest, and dumbest, of all the test doubles. They really just exist to fill in the required interface of the dependency being replaced, and to provide the test with the data ("stubbed values") it needs to execute.

A mock is pre-programmed with expectations about the calls it will receive. If the mock does not receive the expected calls and parameters, it will throw an error, causing the test that called it to fail. Mocks tend not to replicate so much internal logic from their real counterparts as fakes do.

A spy can be thought of as a lightweight mock. All it does is remember what calls it has received, and it can make that information available to the test for assertion purposes. It differs from a mock in that it does not make any assertions itself — it just gathers data about its invocations, against which assertions can be made by the test code. A classic use case for a spy would be to record how many messages were sent to an email service.

Finally, a dummy refers to any test object or value that is used in tests but is never actually inspected by the tests. Dummies are commonly used to stand-in for function parameters. They are usually primitive values or plain objects; at most they will be very lightweight fakes.

The word "dummy" is also used in the context of test data. Dummy data is any data that is injected into a system in place of production data in non-production environments — including, but not limited to, test environments.

There are a few variations on these types of test doubles. For example: a partial mock is backed by a real object (it mocks some, not all, of the methods of the real object it is replacing); a capture replay mock records real API interactions which can then be played back in subsequent tests; an approval mock (aka snapshot mock) captures the actual response and uses that "approved snapshot" in future tests, flagging any deviations for review; an auto-generated contract stub is automatically generated from its contract/interface; and a self-initializing fake is automatically generated by capturing responses from interactions with the real object it is replacing.

The words "mocks", "fakes", and "stubs", although conceptually distinct, are often used interchangeably — even by testing and mocking frameworks. "Mock" is the most misused, and is widely used — incorrectly — as a catch-all term for any kind of test double. The concept of "mock objects" was defined in a 2000 paper by Tim Mackinnon, Steve Freeman, and Philip Craig, and the pattern is specifically described as objects that "replace domain code with dummy implementations that both emulate real functionality and enforce assertions about the behaviour of our code." The objective of the mock object pattern was to remove assertions out from production code — a conventional technique in unit testing at the time — and move them into the test code.

In practice, many test doubles have characteristics of multiple types of doubles. For example, a fake implementation may also have some mock-like behavior, where it asserts that certain methods are called with specific parameters.

In a bid to try to clear up the confusion, Gerard Meszaros coined the term "test double" in his 2007 book xUnit Test Patterns. It is a deliberately generic term intended to encompass all types of test doubles.

If in doubt, prefer the term "test double" over any of the more specific terms!

Trade-offs

One of the most important choices in the design of automated tests is how and when test doubles will be used to stand-in for real dependencies. As with all design decisions, there are trade-offs to consider.

Knowing what to mock, and what not to, is quite subjective. People’s views on the optimal balance changes with context and experience. The following are some general guidelines to help you think through the trade-offs.

The main reasons for using test doubles are:

  • To isolate specific behaviors to test.
  • To increase the speed of test execution.
  • To remove dependencies on external systems, such as databases, that are unavailable in test environments.

The main trade-offs of using test doubles are:

  • Increased complexity of the tests.
  • More scaffolding code.
  • Reduced readability of test code.
  • Increased maintenance costs.

But the biggest issue is increased brittleness. The overuse of doubles can lead to brittle tests. A brittle test is a test that fails, not for real problems that would happen in production, but because the test itself is fragile.

The reason is that the implementations of test doubles may diverge from the real implementations over time. If this happens, the tests may continue to pass, but they will no longer be testing the real behavior of the system-under-test. As bugs increasingly slip through, confidence in the tests will drop. And if the tests are increasingly untrusted as a reliable measure of the correctness of the system, this will decrease confidence in development efforts to introduce new features and other changes.

Brittle tests tend to be the outcome of putting too much implementation detail into the tests themselves. Test doubles are normally the culprit. The bigger the test double — the more details from the real implementation that are embedded in it — the more likely it will diverge from the real implementation over time. Thus, fakes are more prone to drift than mocks, while stubs and spies tend to be more stable.

For these reasons, best practice is to err on the side of high fidelity tests with minimal mocking. We use the term fidelity to refer to how closely the behavior of a system-under-test matches its production behavior. High-fidelity testing is the goal. This means as few dependencies as possible are mocked. Real dependencies — including vendor components that may be installed via a package manager, for example — are preferred over doubles in tests. When dependencies need to be mocked, lightweight doubles, which do not replicate much of the logic of their real implementations, are preferred.

Real dependencies SHOULD be replaced with doubles only to overcome specific problems associated with using real dependencies in test environments — for example if a real implementation is slow, unreliable, non-deterministic, or difficult to instantiate (it requires a network connection, say).

Fakes should be used sparingly, and where they are required — for example, to swap a database abstraction for a simple in-memory store — they should be maintained alongside the real implementations. This will increase the chances of the fakes being kept up-to-date with the real implementations. It also has the benefit of keeping the test code cleaner; there will be less boilerplate in test scripts for the construction of fakes.

Fakes are, to all intents and purposes, real implementations. They actually work. It’s just that they’re optimized for non-production environments. But they very much belong with the rest of the application code, not the test code. Indeed, fakes should have their own tests!

Ideally, the only components that will be swapped for fakes in test scenarios will be those that communicate with external systems such as file systems, databases, and remote services — anything that is not available (or is unreliable) in test environments.

Concluding remarks

It might be tempting to make automated tests as fast and as lightweight as possible, especially at the unit level, by using lots of test doubles. But this is an anti-pattern. It gives a false sense of dependability in the tests. And tests must, above all, be dependable.

If performance of your test suite’s execution becomes an issue, try to adjust your test setup — eg. enable greater parallelization of test execution — before resorting to lowering the fidelity of the tests.

Test-driven development

Every time you encounter a testability problem, there’s an underlying design problem.

– Michael Feathers

Test-driven development (TDD) is a method for writing software by writing tests first, then writing the code that makes the tests pass.

TDD follows a short, repeating cycle known as red-green-refactor:

  1. Red — write a small test for the next piece of desired behavior. Run it. It fails, because the behavior does not exist yet.
  2. Green — write the simplest code that makes the test pass. Nothing more.
  3. Refactor — improve the structure of the code (and the test) while keeping all tests passing.

Each iteration of this cycle adds one small increment of behavior and one small increment of design. The result is that the system is built in a series of tiny, verified steps.

TDD puts you in the position of being a consumer of your own code. When you write a test, you’re effectively writing a client for the component you are about to build. Writing the test first gets you thinking about the design of the component’s interface — how it will be used — before you commit to an implementation.

The idea is that you are more likely to design a good interface if you shift-left your experience of using that interface. For this reason, TDD might be better described as test-driven design.

TDD is most commonly used in bottom-up design processes, in which a system is built in a piecemeal fashion, component by component. However, it can also be effective in top-down design. Indeed, it can be desirable to take a test-driven approach at all test levels: unit, integration, and system.

TDD is the RECOMMENDED approach to writing software. Code that is test-driven tends to be more modular, more maintainable, and easier to change. Non-TDD approaches often result in more tactical and complex tests, tightly coupled to the implementation.

You can’t inspect quality into a product. You must build it in.

– W. Edwards Deming

Worried that TDD will slow down your programmers? Don’t. They probably need slowing down.

– J. B. Rainsberger

That said, TDD has trade-offs. Its tight cycle of small increments can sometimes work against design: with many tests in place, large-scale refactoring becomes harder, because every structural change requires updating both the code and its tests. TDD is most valuable when the design direction is reasonably clear. When the design is still in flux — during early exploration or prototyping — it can be more practical to experiment with different approaches first and write tests once the design stabilizes.

The real objective is what Martin Fowler called self-testing code: a codebase in which you can confidently verify correct behavior with a single command. TDD is the recommended means of getting there, but the end is more important than the means. Developers SHOULD default to writing tests before code, while recognizing that this is a guideline, not a dogma.

Test design

Writing tests is not enough. The tests themselves must be well-designed. Poorly written tests are slow, fragile, hard to understand, and difficult to maintain. Over time they become a burden rather than an asset, and teams stop trusting or running them.

Good test design is governed by the same principles as good application code design: expressiveness, simplicity, and maintainability. The guidelines in this section apply to individual test cases regardless of their type or level.

FIRST principles

The FIRST acronym summarizes five properties that every well-designed test should have.

Fast. Functional tests should run quickly. A functional test suite that takes a long time to execute will be run less frequently, reducing its value as a feedback mechanism. Unit tests in particular should run in milliseconds. Slow tests are usually a sign of unnecessary I/O, real network calls, or excessive setup — all of which should be replaced with test doubles or eliminated through better design.

Independent. Each test should be independent of all others. Tests must not rely on shared mutable state or on a specific execution order. A test that passes only when run after another test, or that fails when run in isolation, indicates hidden coupling. Independence makes it possible to run any subset of tests in any order, to parallelize execution, and to diagnose failures in isolation.

Repeatable. A test should produce the same result every time it is run, in any environment. Tests that pass on one machine and fail on another — or that fail intermittently due to timing, randomness, or external dependencies — undermine confidence in the test suite. Eliminate non-determinism by controlling time, seeding randomness, and replacing external dependencies with test doubles.

Self-validating. A test must have a clear, automated pass/fail outcome. It should not require a developer to manually inspect output, logs, or a database to determine whether the test passed. The assertion within the test is the declaration of expected behavior; if it passes, the test passes; if it fails, the test fails. No human interpretation should be necessary.

Timely. Tests should be written at the same time as — or before — the code they verify. Tests written long after the fact are harder to write well, because the code may not have been designed with testability in mind. Writing tests early (or first, as in test-driven development) keeps code testable and ensures that tests actually reflect the intended behavior.

Readability

A test case is also a specification. It documents the expected behavior of the code under test. For this documentation to be useful, tests must be as readable as any other form of documentation.

Give tests descriptive names that read as a statement of expected behavior: calculatesTaxAtStandardRateForDomesticOrders, not testCalc. The name should make the intent immediately clear without reading the test body.

The Given/When/Then pattern

Structure each test consistently around three phases, in order:

  1. Given — establish the preconditions. Create or configure the component under test and its dependencies, including any test doubles and their canned responses.
  2. When — invoke the single behavior under test. This phase should be a single statement or a small block of calls that exercises the one action the test is verifying.
  3. Then — verify the expected outcomes. Assert the returned values, the state changes, and the interactions that should (or should not) have occurred with any test doubles.

This is the same pattern as arrange–act–assert (AAA); the Given/When/Then labels originate from behavior-driven development (BDD) and read more naturally as prose. Keep each phase visually distinct, using blank lines or comments to separate them if the test body is more than a few lines long. A test that follows this structure reads top-to-bottom as a short narrative: here is the starting state, here is what happened, here is what we expect.

public function testCannotReadReport(): void
{
    // Given
    $this->user->shouldReceive('can')
        ->once()
        ->with('read_own_report')
        ->andReturn(false);

    // When
    $actual = $this->policy->canRead($this->user, $this->report);

    // Then
    $this->assertFalse($actual);
}

The When phase SHOULD contain exactly one action. A test with multiple actions in its When phase is testing more than one behavior and should be split. The Given phase SHOULD contain only the setup that the test genuinely needs; shared, reusable setup belongs in a fixture or factory, not repeated in every test.

Avoid introducing logic into tests — conditionals, loops, and helper computations make tests harder to read and introduce the risk of bugs in the test code itself. If a test requires complex setup, extract it into a clearly named fixture or factory rather than embedding it inline.

One assertion per test (as a guideline)

Tests should be focused. A test that asserts many things at once is hard to read, and when it fails, it is not immediately clear which assertion failed or which behavior is broken.

As a guideline, aim for one logical assertion per test case. This does not mean exactly one assert call — a single logical assertion may require several related checks — but it does mean that each test should verify one distinct aspect of behavior.

Where a single piece of behavior genuinely produces multiple observable outcomes, it is acceptable to assert all of them in one test. The spirit of this guideline is focus and clarity, not mechanical adherence to a line count.

Test architecture

Executable specifications (see BDD) are most powerful when they are built on a layered architecture that separates the specification from the mechanics of driving the system-under-test. This separation is what allows the same specification to remain valid even if the underlying implementation, or even the interface, changes completely.

The four layers

A well-architected suite of executable specifications has four distinct layers:

  1. Test cases — the executable specifications themselves, written in the language of the business domain. A test case reads as a statement of what the system does, eg. shouldBuyBookWithCreditCard.
  2. Domain-specific language (DSL) — an API, internal or external (such as Gherkin), that expresses operations in the vocabulary of the problem domain, eg. searchForBook, addToCart, checkoutWithCreditCard. The DSL is reused across many test cases, and is itself ignorant of the system-under-test.
  3. Protocol drivers — the layer that translates DSL calls into real interactions with the system-under-test: UI automation, API calls, message publishing, or whatever is the natural interface for the system. This is the only layer that has any knowledge of how the system is actually driven.
  4. System-under-test — the running application, including any test data or fixtures it depends on.
Test cases (executable specifications)
    ↓
DSL (problem-domain vocabulary)
    ↓
Protocol drivers (UI, API, messaging, ...)
    ↓
System-under-test

Only the protocol driver layer knows about the system’s actual interface. The test cases and the DSL are completely ignorant of it. This is what makes the architecture powerful: the same executable specifications can be reused across different interfaces (eg. a web UI and a CLI to the same system), and a change to the system’s interface requires updating only the protocol drivers, not the specifications themselves.

The abstraction check

A useful test for whether a specification is written at the right level of abstraction: imagine the same behavior implemented through a completely different kind of interface — a voice-activated system, or a thought-controlled one. Would the specification still make sense?

A specification that says:

Given I am an authorized user
When I enter "scott" into the username field
And I enter "abc123" into the password field
And I click "login"
Then I should be directed to the application landing page

…​fails this test. It describes a UI design, not a behavior — it says nothing about what the user wants, only what one particular interface gives them. If authentication were done by passkey, biometrics, or voice, this specification would need to be rewritten entirely.

A specification written at the correct level of abstraction survives the change of interface:

Given an active, verified user
When the user authenticates
Then the user should be authenticated

This version is agnostic to the authentication mechanism, and remains valid regardless of how the interface evolves.

Apply this check whenever writing or reviewing executable specifications. Implementation details — field names, button labels, UI interactions — are a sign that a specification is coupled to a protocol driver rather than describing a domain behavior.