7 Load Testing Mistakes That Give You False Confidence

A load test that produces false confidence is worse than no load test. At least without a load test, you know you have not validated performance. A load test with fundamental methodological flaws gives engineering teams a false sense of security - they believe they have validated their system when they have actually validated nothing.

These are the seven mistakes we see most often when reviewing engineering teams’ load testing practices. Each one corrupts results in a specific, predictable way.

Mistake 1: Running Tests from a Single Machine

The problem: A single load generator machine has finite resources: CPU, memory, network connections, and bandwidth. When a test machine saturates its own resources, it cannot generate more load - but neither you nor the monitoring you are watching will tell you this is what is happening. The test reports normal-looking numbers while silently throttling itself.

The symptom: throughput plateaus and does not increase even as you raise virtual user count. p95 latency looks suspiciously stable. The test “passes” because the system under test was never actually stressed.

Real-world example: An e-commerce team ran a load test claiming their checkout flow handled 1,000 concurrent users. The test used a single t3.medium EC2 instance as the load generator. The t3.medium has 2 vCPU and 4GB RAM - barely enough to simulate 300 concurrent users with meaningful connection overhead. The test plateaued at ~300 effective users and the team never knew.

The fix: Monitor load generator resource utilization during tests. CPU should stay under 70%, memory under 80%, and network utilization well below interface capacity. For tests requiring more than 500 virtual users, use distributed load generation:

• k6: Use k6 Cloud or run multiple k6 instances behind a k6 operator on Kubernetes
• Locust: Use the built-in master-worker architecture with multiple worker instances
• Rule of thumb: One c5.xlarge AWS instance can reliably generate 1,000-2,000 VUs with k6; scale proportionally for more

Mistake 2: No Think Time Between Requests

The problem: Real users do not send 100 HTTP requests per second from a single browser. They click a link, read the page for 10-30 seconds, click another link, fill out a form, submit. The gap between requests - think time - is fundamental to realistic load simulation.

Scripts without think time generate artificial load that saturates the system under test in unrealistic ways. The test creates 10x more database connections than real users would, exhausts thread pools that would not be exhausted under real load, and generates request patterns that do not match production.

Real-world example: An API serving 500 concurrent real users was tested with 500 virtual users and zero think time. The test generated 15,000 requests/second - 30x what 500 real users would generate. The system “failed” the test. The team added expensive infrastructure to handle the artificial load. Real users at 500 concurrent would have been fine.

The fix: Add realistic think time between requests. Pull session data from your analytics to understand actual user pacing. A common pattern:

// k6 example with realistic think time
export default function() {
  // User arrives at product page
  http.get('/products/featured');
  sleep(3 + Math.random() * 7);  // Reads for 3-10 seconds

  // User views a specific product
  http.get('/products/123');
  sleep(5 + Math.random() * 15); // Considers purchase for 5-20 seconds

  // User adds to cart
  http.post('/cart', { product_id: 123, quantity: 1 });
  sleep(1 + Math.random() * 3);  // Brief pause

  // User checks out
  http.post('/orders', { cart_id: 'abc' });
}

Mistake 3: Testing Only Happy Paths

The problem: Load tests that simulate only successful user journeys miss the failure handling code paths that can be the most expensive to execute.

Real production traffic includes: failed authentication attempts, malformed requests, expired sessions that require token refresh, searches that return no results, 404 errors from invalid URLs, validation errors from form submissions. These error paths often execute more code than happy paths - they query the database to verify the resource exists before returning 404, they log extensively, they send error tracking events.

Real-world example: An authentication service performed fine under happy-path load testing. In production, a wave of automated bot traffic attempting credential stuffing attacks (failed authentication) caused the auth service to execute expensive password hashing operations and database lookups for invalid credentials. The load test had never tested this path. The service fell over.

The fix: Include realistic error distribution in your test scripts. Analyze your production traffic logs for error rates and error types, then include them proportionally in your tests:

// Include a realistic mix of request outcomes
export default function() {
  // 85% valid logins, 15% invalid credentials
  const email = Math.random() < 0.85
    ? `valid-user-${__VU}@example.com`
    : `invalid-user-${__ITER}@notexist.com`;

  http.post('/auth/login', JSON.stringify({ email, password: '...' }));
}

Mistake 4: Using Unrealistic Test Data

The problem: Tests that use the same user ID, product ID, or data set repeatedly do not reflect production behavior. Databases cache frequently accessed rows. Application caches store frequently requested objects. Tests that repeatedly access the same data hit warm caches and show artificially fast response times.

The specific failure: your test shows 50ms API response times. In production, users access diverse data sets that do not fit in cache. Real response times are 300ms because the database actually reads from disk for most requests.

Real-world example: A SaaS team load-tested their analytics dashboard endpoint with the same five user IDs. The database query results were cached in Redis within the first few requests. Every subsequent request hit the cache. Test results showed 10ms response times. Production response times were 800ms because real users access their own unique datasets, almost none of which are in cache.

The fix: Use diverse, realistic test data. The number of unique data entries in your test should be at least 10x your peak virtual user count.

// Use SharedArray to load realistic test data
import { SharedArray } from 'k6/data';

// Load 1000 unique user records for testing
const testUsers = new SharedArray('users', function() {
  return JSON.parse(open('./data/test-users.json'));
});

export default function() {
  // Pick a different user for each VU iteration
  const user = testUsers[(__VU * __ITER) % testUsers.length];
  // This ensures diverse data access across the test
}

Mistake 5: Testing in the Wrong Environment

The problem: Staging environments are systematically different from production in ways that make test results unreliable as predictors of production behavior.

Common differences:

• Smaller instance types (staging c5.large vs production c5.2xlarge)
• Smaller datasets (staging: 50,000 rows vs production: 5,000,000 rows)
• Different connection limits (staging: max_connections=100 vs production: 500)
• Missing infrastructure (no CDN, no read replicas, different cache sizing)
• Stale database statistics (query planner makes different decisions)

The result: tests in staging either under-predict problems (because production has more resources) or over-predict problems (because staging has less data and behaves differently than production under high load).

The fix: Use environment-specific acceptance criteria. If staging runs at 50% of production capacity, expect 50% throughput. Document the configuration differences between staging and production and account for them explicitly.

For critical tests (pre-launch, capacity planning), run against production with a small blast radius:

• Use production infrastructure with a traffic shadowing approach
• Test during low-traffic periods
• Use feature flags to direct a percentage of test traffic to a canary deployment

Mistake 6: Ignoring the Ramp-Up Period

The problem: Starting a load test at full concurrency does not give the system time to warm up. JVM-based applications have JIT compilation that dramatically improves performance after the first few minutes. Application caches start empty. Database query plan caches are cold. Connection pools need time to fill.

Tests that measure performance during the ramp-up period include artificially poor results (cold start performance) in their statistics. The average latency is artificially high; the reported “failures” may be cold-start artifacts, not real performance problems.

Real-world example: A Java microservice showed 10% error rate in the first minute of every load test. The team investigated repeatedly and found nothing wrong. The issue: JVM cold start during the first minute generated timeouts while the JIT compiled hot paths. After 60 seconds, performance was fine.

The fix: Always ramp up gradually, and exclude ramp-up data from your final analysis. In k6, use the startVUs option and ramp from zero:

export const options = {
  stages: [
    { duration: '3m', target: 100 },  // Ramp up - warm up period
    { duration: '10m', target: 100 }, // Steady state - this is what you measure
    { duration: '2m', target: 0 },    // Ramp down
  ],
  // The thresholds apply to the entire test, but you can configure
  // your monitoring to only alert on the steady-state portion
};

For JVM applications, a 3-5 minute warm-up period before your test begins is worth the time.

Mistake 7: Reporting Averages Instead of Percentiles

The problem: Average (mean) latency is a misleading metric. Averages are dominated by the large number of fast requests and hide the tail behavior that real users experience as poor performance.

Consider this distribution: 95% of requests complete in 50ms, 4% complete in 500ms, 1% complete in 8000ms. The average is: (0.95 * 50) + (0.04 * 500) + (0.01 * 8000) = 47.5 + 20 + 80 = 147.5ms. The average looks reasonable. But 1% of users are waiting 8 seconds - at 1 million requests/day, that’s 10,000 users experiencing 8-second wait times every day.

The fix: Report and set thresholds on percentiles, not averages. The standard reporting set:

• p50 (median): typical user experience
• p95: upper bound for most users
• p99: tail behavior, important for high-volume services
• p99.9: extreme tail, relevant when 0.1% represents many users

// Setting meaningful threshold on percentiles, not averages
export const options = {
  thresholds: {
    // GOOD: percentile-based thresholds
    'http_req_duration': ['p(95)<500', 'p(99)<2000'],

    // BAD: average-based threshold (never use this)
    // 'http_req_duration': ['avg<200'],
  },
};

If your team or stakeholders are asking for average response time, educate them on why percentiles matter more. “Our average response time is 150ms” sounds good. “1% of our requests take over 5 seconds” tells a very different story.

The Meta-Mistake: Tests Nobody Acts On

There is a mistake that supersedes all seven above: running load tests and not acting on the findings.

Teams fall into this pattern when:

• Load test results are generated but not reviewed
• Findings are captured in a document but not triaged into engineering tasks
• Engineering tasks are created but deprioritized indefinitely

A load test with excellent methodology that produces no engineering action is worth exactly as much as no load test at all.

The fix: Establish a clear action protocol for load test findings:

1. Every load test result is reviewed by at least one engineer within 24 hours
2. Any threshold violation or significant performance regression becomes a priority bug, not a backlog item
3. The engineer who introduced the performance regression (identifiable from git blame) is assigned the fix
4. A follow-up load test is run to confirm the fix before closing the ticket

Load testing only improves your system’s performance if the findings drive engineering changes. Build that feedback loop into your process from the start.

If your team wants to build a load testing practice that actually produces results, our load testing program setup covers tooling, methodology, CI/CD integration, and the process changes that make findings actionable.