What Is Soak Testing? How to Run One and What It Actually Finds

Soak testing runs your system at moderate, realistic load for hours or days to find the failures that only show up with time. A load test asks whether you survive peak traffic for an hour. A soak test asks whether you are still healthy on day three. Those are different questions, and passing the first tells you very little about the second.

The distinction matters because the failures are structurally different. Load tests find capacity problems: too many concurrent users, not enough CPU, a database that cannot keep up. Soak tests find accumulation problems: something that grows a little on every request and eventually runs out. You cannot find an accumulation problem by adding concurrency, only by adding time.

What soak testing actually is

Soak testing, also called endurance testing, holds load steady at a level your system should comfortably handle and keeps it there for an extended run. The load level is deliberately unremarkable, usually somewhere around normal production traffic rather than peak. The duration is the variable under test.

This trips people up, because it looks like a weak test. The dashboard is boring for the first two hours. That is the point. You are not trying to break the system with force; you are waiting for it to break itself.

Soak testing vs load testing vs stress testing

The three are often used loosely, so it is worth being precise about what each one varies:

A system can pass every load test you throw at it and still fall over on a Thursday afternoon because a connection pool leaks a handful of handles per hour. Load testing sizes your capacity. Soak testing proves you can hold it.

What a soak test actually finds

In practice the findings cluster into two groups. The first is genuine leaks:

• Memory leaks. Heap or RSS climbing steadily across the run. The classic case is a cache or listener registry that gets appended to and never pruned.
• Connection leaks. Database or HTTP connections acquired and not returned. Pool utilisation creeps toward its ceiling, then requests start queueing for a connection that never frees.
• File descriptor and socket leaks. Sockets left in CLOSE_WAIT, files opened in an error path that skips the close. Eventually you hit the process limit and everything fails at once.
• Thread pool exhaustion. Threads parked on work that never completes, so the pool has nothing left to schedule with.
• Unbounded caches. Not strictly a leak, but the same shape: an in-memory map with no eviction policy that grows with cardinality you did not anticipate.

The second group is the more interesting one, because these are not leaks at all and they are invisible to every other kind of test:

• Scheduled jobs competing with live traffic. A nightly batch, a backup window, or a reindex that halves your throughput for 40 minutes. You only see it if your test is running when the job fires.
• Credential and certificate expiry. A token with a 12-hour lifetime that the client never refreshes. A four-hour test passes; a 24-hour test does not.
• Log rotation and disk growth. Logs filling a volume, or a rotation that briefly blocks writes and shows up as a latency spike at exactly the same minute every hour.
• Query plans degrading as tables grow. If your test writes data, the database it is querying at hour 20 is not the database it queried at hour one.

That second group is the strongest argument for running soak tests at all. Nothing else in a normal performance testing practice will catch them.

How long to run

Long enough that your slowest accumulation cycle has time to show a trend. Rather than picking a number, work from the system:

• Run longer than your longest cache TTL, so you see at least one full expiry and refill.
• Run longer than your session or token lifetime, so refresh logic is exercised rather than assumed.
• Run across at least one full cycle of every scheduled job - batch, rotation, backup, certificate refresh.
• If pods are recycled by deploys every day, an 8-hour run is often enough, because the restart masks anything slower. If processes routinely live for weeks, run for days, not hours.

The honest default for a system you cannot trivially restart is 24 hours, extended to 72 when you have had a production incident you could not explain.

What to monitor, and how to read it

The single most important habit in soak testing is to read the slope, not the value.

Track heap and RSS memory, open file descriptors and socket counts, connection pool utilisation for every pool, thread counts, garbage collection frequency and pause duration, disk usage on log and temp volumes, and p95 and p99 latency plotted across the entire run rather than aggregated into a single summary number.

Then judge them on trend:

• Flat is a pass. Memory that rises during warm-up and then plateaus is healthy.
• A steady climb is a finding, even if the value is still well inside its limit when the test ends. A heap growing 2% an hour is fine for a day and fatal in a week. The test does not need to reach the failure; it only needs to establish the gradient.
• A sawtooth is usually fine. Memory that climbs and drops sharply is garbage collection doing its job. Watch whether the floor of each sawtooth is rising - that is the leak hiding inside normal-looking behaviour.
• Step changes point at events, not accumulation. A sudden jump at 02:00 is a scheduled job, not a leak. Correlate against the clock before assuming it is your code.

This is also why a soak test with only a pass or fail verdict is close to useless. The deliverable is the trend chart.

Setting pass and fail criteria

Write them before the run, the same way you would for a load test. Because the signal is a gradient, the criteria should be expressed as rates rather than absolutes:

• Heap or RSS growth stays below a defined percentage per hour after warm-up.
• Connection pool utilisation does not trend upward across the run.
• Open file descriptors return to baseline between traffic cycles.
• p99 latency in the final hour is within a defined percentage of the first full hour after warm-up.
• Error rate stays flat rather than climbing, however small the absolute number.
• No unplanned restarts, and no out-of-memory kills.

That last one deserves emphasis. If a container was restarted mid-run by the orchestrator, your memory chart may look beautiful and mean nothing, because the counter reset. Always check restart counts before you trust a clean result.

Common mistakes

• Running at peak load. A soak test at maximum concurrency is just a long stress test, and it usually fails for capacity reasons before any accumulation problem becomes visible. Use realistic, moderate load.
• Restarting between phases. Any restart resets the very counters you are measuring. The run must be continuous.
• Testing against a freshly provisioned environment every time. Some problems only appear once there is data in the database and entries in the caches.
• Only aggregating results. A single p99 for the whole run averages away the fact that it doubled in the last two hours, which was the entire finding.
• Ignoring the infrastructure. Disk and file descriptors are where soak tests most often fail, and they are the metrics application dashboards most often omit.
• Not running long enough to reach the cycle that matters. A six-hour test on a system with an eight-hour token lifetime proves very little.

Where soak testing fits

Monthly is a reasonable cadence for a system under active development, plus an extra run before any release that changes connection handling, caching, session management, or a major dependency version. Because the wall-clock cost is high and the attention cost is low once automated, the practical pattern is a scheduled overnight or weekend run against a persistent staging environment, with the trend charts reviewed the next working day.

Keep the short load test as your per-release gate and the stress test as your quarterly capacity check. The soak test is the one that tells you whether what you built can simply keep running.

If you want this run properly against your own stack rather than built from scratch, our soak testing service covers 8 to 72-hour endurance runs with leak detection, trend analysis, and a report that names the component rather than just the symptom. Talk to us about your system.