TS-14: Performance Testing

This technical standard covers testing strategies that evaluate the quality attributes of a software system — its speed, capacity, scalability, security, accessibility, usability, compliance, and resilience.

These quality attributes are also known as non-functional requirements (NFRs) or cross-functional requirements (CFRs). They are distinguished from functional requirements in that they describe how well the system performs its functions, rather than what functions it performs. Note that performance in the strict sense — speed and capacity — is just one of these quality attributes; despite this standard’s title, its scope is the qualities as a whole, not performance alone.

Non-functional requirements are architecturally significant. They cannot be tested as an afterthought or addressed as a final quality gate before release. They influence design decisions at every level of the system, from infrastructure and data storage through application logic and user interfaces. This standard therefore begins with the principle of shifting left — integrating non-functional testing early and continuously throughout the development lifecycle.

See also TS-13: Functional Testing. TS-10: Releasing covers canary testing and other release testing strategies.

Shift left

Too often, performance testing is treated as a last-mile quality gate before release. This approach carries enormous risks. By the time performance problems are discovered late in development — in the worst cases, only after changes are deployed to production — the remediation effort can be orders-of-magnitude greater than if they were discovered earlier. Quality requirements — of which performance in the strict sense is one — are architecturally significant, cross-cutting concerns that influence design choices at every level of the software stack, from infrastructure and data storage through application logic and user interfaces.

Shifting left means integrating performance evaluation early in the software development lifecycle — treating it as a continuous activity rather than a final checkpoint. By testing performance during the design and development phases, teams can identify bottlenecks, scalability limitations, and resource constraints when they are far easier and cheaper to address. Early detection prevents the cascading effects of poor performance choices that become deeply embedded in the application architecture.

This same principle — shifting left — applies to all non-functional testing, not just performance. Security vulnerabilities, accessibility barriers, and compliance gaps are all significantly cheaper to address when they are caught during development rather than discovered in production or, worse, by an external audit or a user complaint.

If you want to make a fast program, use a slow computer.

Moreover, shifting left fundamentally changes team culture and accountability. When developers receive immediate feedback on the non-functional impact of their code — through automated performance tests in CI/CD pipelines, security scanning in pull requests, or accessibility checks in the development environment — they become more conscious of these quality attributes in their daily work. Non-functional quality becomes a shared responsibility, not a separate team’s concern.

Best practices:

• Integrate non-functional tests — performance, security, accessibility, compliance — into the CI/CD pipeline so they run automatically on every change, not just before releases.
• Establish non-functional requirements and acceptance criteria at the start of the project, alongside functional requirements. Performance targets, security standards, and accessibility conformance levels should be defined before implementation begins.
• Provide developers with tools to evaluate non-functional characteristics locally, before code is committed. Fast feedback loops are essential for shifting left to be practical.
• Treat non-functional test failures with the same urgency as functional test failures. A performance regression or a security vulnerability is as much a defect as a broken feature.
• Do not wait for a dedicated "performance testing phase" or "security audit." These activities should be continuous, not periodic.
• Use lightweight, representative performance benchmarks during development, and reserve comprehensive performance testing (with full production-like environments and data sets) for integration and pre-release stages.

Performance testing

Performance testing evaluates whether the system-under-test meets its non-functional requirements under normal operational conditions. Just as with functional requirements, non-functional requirements MUST have well-defined acceptance criteria — typically defined as performance metrics and thresholds such as response time, throughput, and resource utilization.

Performance testing encompasses a range of specific techniques, several of which — capacity testing, scalability testing, and others — are covered in dedicated sections of this standard. This section addresses the general principles and practices that apply across all forms of performance evaluation.

The key performance metrics to evaluate include:

• Response time — how long the system takes to respond to a request, usually measured at specific percentiles (p50, p95, p99) rather than as averages, since averages hide tail latency.
• Throughput — the number of transactions or requests the system can process per unit of time.
• Resource utilization — CPU, memory, disk I/O, and network bandwidth consumed during operation. High utilization under normal load may indicate capacity risks.
• Error rate — the proportion of requests that result in errors under load. A system may appear fast while silently failing a percentage of requests.

Performance tests serve both a verification and a monitoring function. In the verification role, they confirm that the system meets specific performance targets before release. In the monitoring role, they track performance trends over time, providing early warning of gradual degradation — sometimes called performance drift — that might not be caught by functional tests.

Baseline testing establishes reference measurements under known conditions — a defined number of concurrent users, a representative data set, a specific infrastructure configuration. All subsequent performance evaluations are compared against this baseline. Without baselines, it is impossible to determine whether performance is improving or degrading.

Soak testing (also called endurance testing) runs the system under sustained load for an extended period — hours or days rather than minutes. The purpose is to expose problems that only emerge over time, such as memory leaks, connection pool exhaustion, log file growth, and gradual resource depletion.

Spike testing subjects the system to sudden, dramatic increases in load, followed by equally sudden decreases. This reveals how the system behaves when traffic surges unexpectedly and whether it recovers gracefully when the spike subsides.

Best practices:

• Define performance requirements as specific, measurable targets — eg. "p95 response time under 200ms at 500 concurrent users" — not vague aspirations like "the system should be fast."
• Simulate realistic user behavior patterns and transaction volumes. Synthetic benchmarks that do not reflect real usage patterns will give misleading results.
• Conduct performance testing in environments that mirror production infrastructure as closely as possible. Differences in hardware, network topology, or data volume can invalidate results.
• Establish baseline performance metrics early in the project, and track them over time to detect gradual degradation.
• Automate performance tests and integrate them into the CI/CD pipeline, so regressions in response time or throughput are caught early.
• Use percentile-based metrics (p95, p99) rather than averages. A system with an average response time of 100ms may have a p99 of 5 seconds — a serious problem that the average conceals.
• Include soak tests in the regular testing cadence, not just before major releases. Resource leaks often take hours to manifest.
• Monitor all layers of the stack during performance tests — application, database, network, infrastructure — to identify where bottlenecks occur.

Capacity testing

Also known as load testing, capacity testing evaluates whether the system continues to meet its performance requirements under increasing load — and determines the maximum workload the system can handle while maintaining acceptable performance levels.

Capacity is a requirement like any other, and it MUST be specified and tested. Capacity encompasses not only the number of concurrent requests or users the system can support, but also batch record processing volumes, file upload and download sizes, message queue depths, and data storage limits. The purpose of capacity testing is to understand the system’s operational limits before users discover them in production.

Variations on capacity testing include:

• Volume testing evaluates performance when processing large datasets — bulk imports, large query result sets, high-volume event streams, or databases that have grown significantly beyond their initial size. Volume testing reveals whether the system’s data handling remains efficient at scale.
• Stress testing pushes the system beyond its designed operational capacity. The purpose is not to verify that the system works under overload — it is expected that it will not — but to understand how it fails. Does it degrade gracefully, shedding non-essential work while maintaining core functionality? Or does it fail catastrophically, corrupting data or becoming unresponsive? Stress testing reveals which components are the most sensitive to increased load and should be prioritized for optimization.
• Rate-limit testing verifies that the system correctly enforces its configured rate limits. For API providers, this means confirming that clients exceeding defined request thresholds receive appropriate throttling responses (typically HTTP 429). For API consumers, it means understanding how the system behaves when it is rate-limited by upstream services. Achieving consistent results at high request rates often requires distributed load generation, since a single test machine may not produce sufficient concurrency due to constraints such as available CPU cores and network latency.

Best practices:

• Establish baseline performance metrics under normal conditions before beginning capacity testing. Without baselines, capacity test results have no meaningful reference point.
• Increase load incrementally to identify the precise points where performance starts to degrade. Sudden jumps from low load to extreme load make it difficult to pinpoint the threshold.
• Design test scenarios that reflect anticipated growth in users, transactions, or data volumes. Test against the projected load for six months or a year ahead, not just today’s load.
• During capacity testing, monitor all system components — application servers, databases, caches, message queues, and network infrastructure. This helps identify bottlenecks and system limits, which may appear in unexpected places.
• Test recovery behavior after periods of overload. Verify that the system returns to normal performance levels when load is reduced, without requiring manual intervention.
• For rate-limit testing, verify both the enforcement of limits and the behavior of the system when limits are reached. Error responses should be informative, and the system should recover immediately once the rate drops below the threshold.
• Capacity test results should feed directly into infrastructure planning and auto-scaling configuration.

Scalability testing

Scalability testing evaluates the system’s ability to accommodate increasing workloads by adding resources — and to release those resources when demand subsides. Where capacity testing determines the limits of a fixed configuration, scalability testing determines how effectively the system can grow beyond those limits.

There are two fundamental approaches to scaling:

• Horizontal scaling (scaling out) — adding more instances of the application to distribute load. This approach is RECOMMENDED for web services and cloud-native applications because it offers theoretically unlimited growth and avoids single points of failure.
• Vertical scaling (scaling up) — increasing the resources (CPU, memory, storage) available to a single instance. Vertical scaling is simpler to implement but has hard limits defined by the available hardware, and provides no redundancy.

Most modern systems are designed for horizontal scalability, but few are truly scalable in practice without deliberate architectural choices. State management, session affinity, database contention, and shared resource locking can all prevent a system from scaling horizontally, even when the infrastructure supports it.

Scalability testing evaluates several dimensions:

• Provisioning speed — how quickly new instances can be brought online when demand increases. If auto-scaling is configured, this is the time lag between a spike in demand and the availability of additional capacity. A system that takes ten minutes to scale while traffic doubles every two minutes will still suffer outages despite having auto-scaling in place.
• Scaling efficiency — whether adding resources produces a proportional increase in capacity. If doubling the number of instances increases throughput by only 30%, the system has a scaling bottleneck — often in shared infrastructure such as databases, caches, or message brokers.
• Scale-down behavior — whether the system can safely reduce capacity when demand drops. Poorly designed scale-down can interrupt in-flight requests, break active connections, or leave orphaned resources consuming budget.
• Data scalability — whether the data layer scales alongside the application. Adding application instances is futile if all of them contend for the same database connection pool. Data partitioning, read replicas, and distributed caching strategies are common solutions, but they must be tested under realistic conditions.

Best practices:

• Design scalability tests around realistic growth scenarios — projected increases in users, transactions, data volume, and geographic distribution.
• Test auto-scaling configuration by generating load patterns that trigger scaling events, and measure the time from the scaling trigger to the availability of the new capacity.
• Verify scaling efficiency by measuring throughput at increasing instance counts. Plot the results to identify the point of diminishing returns.
• Test scale-down as rigorously as scale-up. Verify that active connections are drained gracefully and that no data is lost during instance removal.
• Include the data layer in scalability tests. The application tier and data tier must scale together; testing only the application tier gives a false picture of the system’s actual capacity.
• Test under conditions that reflect production topology — including network latency between zones or regions, load balancer configuration, and DNS propagation delays.

Security testing

Security testing evaluates the system’s ability to protect data and functionality against unauthorized access, malicious attacks, and unintentional damage. The scope of security testing is broad, encompassing authentication, authorization, data protection, input validation, session management, and vulnerability management. The goal is to identify weaknesses that could be exploited — before an attacker does.

Security is not a feature that can be tested at the end and bolted on afterward. Security requirements are architecturally significant, cross-cutting concerns that influence decisions at every layer of the system — from infrastructure configuration and network topology through to application logic and user interface design. Security testing MUST therefore be integrated throughout the development lifecycle, not reserved for a pre-release audit.

The main categories of security testing include:

• Vulnerability scanning uses automated tools to identify known vulnerabilities in the system’s code, configuration, and dependencies. This includes static application security testing (SAST), which analyzes source code without executing it, and dynamic application security testing (DAST), which probes the running application for exploitable weaknesses. Neither approach alone is sufficient; SAST catches issues in code that may not be reachable at runtime, while DAST catches issues that only manifest during execution.
• Penetration testing is a specialized form of security testing in which testers simulate real-world attacks against the system. Penetration tests go beyond automated scanning by applying creative, adversarial thinking to discover vulnerabilities that automated tools might miss. Penetration testing can be conducted as black-box (no knowledge of internals), white-box (full access to source code and architecture), or gray-box (partial knowledge). It is RECOMMENDED that critical systems undergo regular penetration testing by qualified security professionals.
• Dependency scanning identifies known vulnerabilities in third-party libraries and components. Modern applications rely heavily on open-source dependencies, and new vulnerabilities are disclosed regularly. Automated dependency scanning — integrated into the CI/CD pipeline — ensures that known vulnerabilities are detected promptly.
• Authentication and authorization testing verifies that access control mechanisms work correctly: that users can only access resources they are entitled to, that privilege escalation is not possible, and that authentication cannot be bypassed. This includes testing password policies, multi-factor authentication, token management, and session handling.
• Data protection testing verifies that sensitive data is encrypted in transit (TLS) and at rest, that encryption keys are managed securely, and that data is not inadvertently exposed through logs, error messages, or API responses.

Best practices:

• Conduct security testing throughout the development lifecycle — not just before release. Automated vulnerability scanning should run on every build.
• Use a combination of automated scanning and manual penetration testing. Automated tools provide breadth; manual testing provides depth and creative adversarial thinking.
• Test all access control mechanisms, including authentication, authorization, session management, and API keys. Verify both positive cases (authorized access works) and negative cases (unauthorized access is denied).
• Verify that sensitive data is encrypted in transit and at rest, and that encryption algorithms and key lengths meet current standards.
• Include dependency scanning in the CI/CD pipeline to detect known vulnerabilities in third-party libraries. Establish a policy for how quickly known vulnerabilities must be remediated based on severity.
• Follow established frameworks such as the OWASP Testing Guide and the OWASP Top 10 for structured, comprehensive coverage.
• Verify that error messages and logs do not expose sensitive system internals — stack traces, database schemas, internal paths, or configuration details.
• Engage security specialists for critical systems and for penetration testing. Security testing requires a different mindset from functional testing.

Accessibility testing

Accessibility testing verifies that the system is usable by people with disabilities, including visual, auditory, motor, and cognitive impairments. Accessibility is both a legal requirement in many jurisdictions and a fundamental quality attribute — a system that excludes a significant portion of its potential users is not fit for purpose.

Accessibility testing evaluates compliance with standards such as the Web Content Accessibility Guidelines (WCAG), Section 508 of the US Rehabilitation Act, and the European Accessibility Act. These standards define specific, testable success criteria across multiple conformance levels (A, AA, AAA), with Level AA being the most widely adopted target.

Accessibility cannot be fully evaluated by automated tools alone. Automated scanners can detect many common issues — missing alternative text, insufficient color contrast, missing form labels, incorrect heading hierarchy — but they cannot evaluate the actual experience of using the system with assistive technology. A page may pass every automated check and still be unusable for a screen reader user if the reading order is illogical, interactive elements are poorly labeled, or focus management is broken.

Accessibility testing should address multiple categories of impairment:

• Visual impairments — test with screen readers (eg. NVDA, JAWS, VoiceOver), screen magnifiers, and high-contrast modes. Verify that all content and functionality is available without visual perception.
• Motor impairments — test all functionality using keyboard-only navigation. Verify that focus order is logical, that all interactive elements are reachable, and that there are no keyboard traps.
• Auditory impairments — verify that audio content has captions or transcripts, and that information conveyed through sound is also conveyed visually.
• Cognitive impairments — evaluate the clarity of language, consistency of navigation, predictability of interactions, and availability of error prevention and recovery mechanisms.

Best practices:

• Test with real assistive technologies — screen readers, keyboard-only navigation, voice control, and screen magnifiers — not just automated scanning tools.
• Automated accessibility scanners are a valuable first pass and should be integrated into the CI/CD pipeline, but manual testing with assistive devices is essential for evaluating the actual user experience.
• Include users with disabilities in usability testing where possible. No amount of expert evaluation is a substitute for real user feedback.
• Address accessibility from the start of the design process. Retrofitting accessibility into an existing system is significantly more expensive than building it in from the beginning.
• Establish a target conformance level (eg. WCAG 2.1 Level AA) and treat failures against that level as defects that block release.
• Test across multiple assistive technology and browser combinations. Screen reader behavior varies significantly across platforms.
• Document accessibility requirements and test results as part of the standard quality assurance process, not as a separate activity.

Usability testing

Usability testing assesses the ease of use, intuitiveness, and overall user experience of the system. It evaluates whether users can accomplish their goals effectively, efficiently, and with satisfaction — without extensive training or assistance.

Usability testing is inherently human-centered. It cannot be fully automated, because it depends on observing real users interacting with the system and interpreting their behavior, frustrations, and mental models. Automated tools can measure some usability proxies — page load times, click depths, error frequencies — but they cannot evaluate whether a user feels confident, confused, or frustrated.

Usability is often treated as subjective and therefore untestable. This is a mistake. Usability can and should be measured using concrete metrics:

• Task completion rate — can users actually accomplish what they set out to do?
• Time on task — how long does it take? Improving over time suggests the interface is learnable.
• Error rate — how often do users make mistakes, and how easily do they recover?
• Learnability — how quickly can new users become productive?
• User satisfaction — measured through standardized questionnaires such as the System Usability Scale (SUS).

Usability testing is most effective when conducted early and iteratively. Testing prototypes, wireframes, or partial implementations reveals design problems before they become entrenched in the code.

Best practices:

• Recruit participants who represent actual target users, not developers or testers. Internal team members have fundamentally different mental models and cannot provide representative feedback.
• Observe users completing realistic tasks without providing guidance or assistance. The tester’s role is to observe and record, not to coach.
• Measure both quantitative metrics (task completion rate, time on task, error rate) and qualitative feedback (interviews, think-aloud protocols, satisfaction surveys).
• Test early prototypes and wireframes to identify usability issues before full implementation. Usability problems are far cheaper to fix in wireframes than in production code.
• Conduct usability testing iteratively — test, improve, and test again. A single round of usability testing is not sufficient.
• Do not conflate usability with visual aesthetics. A visually appealing interface may still be difficult to use; a plain one may be highly effective.

Compliance testing

Compliance testing verifies that the system adheres to industry standards, regulatory requirements, organizational procedures, and contractual obligations. The scope varies widely depending on the domain — from data protection regulations (such as GDPR and HIPAA) to industry-specific frameworks (such as PCI DSS for payment processing) to organizational coding standards and operational procedures.

Compliance testing is distinct from other test types in that the acceptance criteria are defined externally — by regulators, standards bodies, or contractual agreements — rather than by the development team or product owners. This imposes a discipline that is often uncomfortable for teams accustomed to defining their own "done" criteria: compliance is not negotiable, and partial compliance is often as unacceptable as non-compliance.

The consequences of non-compliance can be severe. Regulatory violations may result in financial penalties, legal action, loss of operating licenses, or reputational damage. For this reason, compliance requirements should be treated with the same rigor as the most critical functional requirements — failures should block releases, and evidence of compliance should be maintained systematically.

Compliance testing typically covers multiple dimensions:

• Regulatory compliance — verification that the system meets the requirements imposed by laws and regulations applicable to its domain and jurisdiction.
• Standards compliance — verification against industry standards such as ISO 27001, SOC 2, or domain-specific frameworks.
• Contractual compliance — verification that the system meets the specific requirements defined in customer or partner agreements.
• Internal compliance — verification that the system adheres to organizational coding standards, architectural guidelines, and operational procedures.

Best practices:

• Identify all relevant standards, regulations, and guidelines early in the project lifecycle. Compliance requirements should be captured as part of the initial requirements analysis, not discovered during pre-release audits.
• Create compliance checklists mapped to specific requirements, and maintain traceability between those requirements and the test cases that verify them.
• Engage compliance experts or auditors when testing against complex regulations. Development teams should not be solely responsible for interpreting regulatory requirements.
• Document compliance evidence thoroughly for audit purposes. Automated reports from test runs can supplement manual documentation.
• Treat compliance requirements as first-class acceptance criteria. Compliance failures MUST block releases, just as critical functional test failures do.
• Automate compliance checks where possible — particularly for standards compliance and internal coding standards — and integrate them into the CI/CD pipeline.

Recovery testing

Recovery testing verifies the system’s ability to recover from failures — including hardware faults, software crashes, network outages, and data corruption — and resume normal operation with minimal data loss and downtime. This includes validating backup and restore procedures, failover mechanisms, and disaster recovery plans.

Recovery testing is closely related to resilience, and overlaps with chaos testing (covered in TS-13: Functional Testing). Where chaos testing proactively introduces failures to discover unknown weaknesses, recovery testing validates that known recovery procedures work correctly and meet their defined objectives.

Every system that matters will eventually fail. The question is not whether failures will occur, but how the system and the team will respond when they do. Recovery testing turns that response from a hope into a verified capability.

Key metrics in recovery testing include:

• Recovery Time Objective (RTO) — the maximum acceptable time to restore service after a failure. This is a business-driven metric: how long can the organization tolerate the system being unavailable?
• Recovery Point Objective (RPO) — the maximum acceptable amount of data loss, measured in time. An RPO of one hour means the organization can tolerate losing up to one hour of data.

RTO and RPO targets MUST be defined as explicit requirements and tested against. A backup strategy that produces backups every 24 hours cannot meet an RPO of one hour.

Best practices:

• Simulate realistic failure scenarios including power loss, network interruption, storage failures, process crashes, and data corruption.
• Verify that data integrity is maintained during and after failures. Partial writes and in-flight transactions should be handled atomically — the system should not silently lose or corrupt data.
• Test backup and restore procedures regularly under various failure conditions, not just once during initial setup. Backups that have never been tested are not backups — they are assumptions.
• Measure actual RTO and RPO against defined targets, and report deviations as defects.
• Document recovery procedures clearly for operations teams. Recovery procedures should be executable by on-call staff under pressure, not just by the engineers who designed them.
• Test failover mechanisms for systems requiring high availability, including automatic failover, manual failover, and failback. Verify that failover is seamless to users and that no data is lost during the transition.
• Test recovery from partial failures — not just complete outages. A single failed database replica, a lost network link between zones, or a corrupted cache should not require full system recovery.

Installation and compatibility testing

Installation testing validates that installation, upgrade, and uninstallation procedures work correctly across all supported configurations. The goal is to verify that users can successfully deploy and configure the system without encountering errors that block adoption.

Closely related is compatibility testing, which verifies that the application performs correctly across different combinations of hardware, operating systems, browsers, network configurations, and other environmental variables. Where installation testing focuses on the deployment process itself, compatibility testing focuses on post-installation behavior across diverse environments.

Configuration testing is another related concern: verifying that the system behaves correctly under different configuration options, including minimal, recommended, and non-default settings. This includes testing the effect of adding or modifying resources such as memory, disk space, and CPU allocation.

These three types of testing are grouped together because they share a common focus: ensuring the system works correctly not just in the development team’s environment, but in the varied environments where it will actually be deployed and operated.

Best practices:

• Test installation procedures on clean systems representing minimum, recommended, and various realistic deployment configurations.
• Verify that prerequisites are clearly documented and validated during installation. Where possible, the installer itself should check for prerequisites and provide clear feedback when they are not met.
• Test upgrade paths from previous versions, including data migration and configuration preservation. Users should not lose data or settings when upgrading.
• Validate that uninstallation procedures leave the system in a clean state — no orphaned files, services, or configuration entries.
• Document and test configuration options systematically. Ensure installation and configuration logs provide sufficient detail for troubleshooting.
• For compatibility testing, define a support matrix of target environments and test against each combination systematically. Automate cross-browser and cross-platform testing where possible.
• For web applications, test across the full range of supported browsers, operating systems, and device form factors. Rendering differences, JavaScript engine variations, and API availability can all cause compatibility issues.