Testing

Once a guarded call is in your application, your tests have to get through it. This guide covers the sharp edges: keeping tests off the clock, isolating them from each other, and reaching the failure paths on purpose.

ExternalService.Test ships the setup and assertions this guide would otherwise ask you to hand-write — use ExternalService.Test imports them, and each section below uses them where they apply. The explanations stay, because the helpers replace the typing, not the model.

Every example here is executed as part of this library's own suite (test/testing_guide_examples_test.exs), so they compile and their assertions hold.

The one thing to know first: service state is global

A service is not a process you start per test. Its configuration lives in :persistent_term keyed on the service term, and its circuit breaker lives in :fuse under the same key. Two consequences follow, and both bite quietly:

• Nothing is torn down for you. A test that trips the breaker leaves it tripped for every test that runs afterwards.
• async: true tests sharing a service share one breaker and one rate-limit bucket, so they can trip each other, and the failure looks like flakiness.

Everything below follows from this.

Asserting that your configuration works

The tests above check that a call behaves. ExternalService.simulate/3 checks that the configuration behaves — that the breaker opens, that it opens soon enough, and that a dead dependency does not absorb more load than you meant it to:

test "our payments breaker actually opens, and fast enough" do
  assert %ExternalService.Simulation{opens_after: opens, worst_call: worst, attempts: attempts} =
           ExternalService.simulate(MyApp.Payments, :always_failing)

  assert opens <= 5
  assert worst < 2_000
  assert attempts <= 25
end

It runs on a virtual clock, so a configuration that would take two minutes of real failure to trip is simulated in microseconds. Nothing sleeps and no service is started.

The scenario worth adding once you know your dependency is that a slow one is not the same as a failing one:

# Attempts that succeed but take five seconds each. The breaker never opens —
# nothing is failing — so :worst_call is the number that matters.
assert %ExternalService.Simulation{worst_call: worst} =
         ExternalService.simulate(MyApp.Payments, {:slow, 5_000})

Attempt duration is the one thing a configuration cannot state, and the thing that makes a hand-sized :within too narrow. {:always_failing, attempt_ms} is how to find out what your configuration does when attempts are slow and failing.

Isolating tests from each other

The cleanest fix is to give each test its own service term. Service terms are arbitrary — any term identifying the service — so derive one from the test name and tear it down afterwards:

setup context do
  service = :"#{context.module}.#{context.test}"
  on_exit(fn -> ExternalService.stop(service) end)
  {:ok, service: service}
end

ExternalService.stop/1 is idempotent, so this is safe whether or not the test got as far as starting the service.

Distinct terms are genuinely independent — melting one leaves the other untouched:

Enum.each(1..2, fn _ -> ExternalService.CircuitBreaker.melt(one) end)

assert ExternalService.blown?(one)
refute ExternalService.blown?(two)

** (NimbleOptions.ValidationError) unknown options [:name],
   valid options are: [:circuit_breaker, :rate_limit, :retry, :sleep_function]

Where per-test services are impractical — which is most of the time with a front-door module — reset the shared service instead. ExternalService.reset_all/1 clears both stateful mechanisms, so nothing a test does to the breaker or the rate limit budget survives into the next one:

setup do
  MyApp.Stripe.reset_all()
  :ok
end

Use reset_all/1 rather than reset/1 here. reset/1 closes the breaker and deliberately leaves the limiter alone, because clearing a limiter in production releases a burst at the service — so a test that drained the budget would leave the next one throttled.

Resetting shares state rather than isolating it, so it does not make concurrent tests independent: two async: true tests can still interleave a reset with the other's assertions. Which leads to the tiering below.

Which technique for which test

The middle row is the only one that gives up async: true, and it is normally the smallest group — those tests are in-memory and fast, so serializing them costs little.

Keeping tests off the clock

Retries

Retry delays are the usual culprit. base: 0 with a linear backoff removes them entirely:

ExternalService.start(service,
  circuit_breaker: [tolerate: 100, within: :timer.seconds(10)],
  retry: [max_attempts: 3, backoff: :linear, base: 0]
)

{elapsed, result} = :timer.tc(fn -> ExternalService.call(service, fn -> :retry end) end)

assert {:error, %ExternalService.RetriesExhausted{}} = result
assert elapsed < 500_000   # three attempts, and none of them waited

Note how loose that bound is. It is there to separate the backoff was removed from production backoff is still on — an HTTP service's 1.5-second retry window — and not to measure anything. A tight bound on elapsed time measures the machine the suite is running on: this assertion used to be 50_000, and failed on a busy CI runner at 68ms with nothing wrong. If you want to assert on the delays themselves, assert on the delays themselves, with :sleep_function below.

The generous :tolerate is deliberate: a test that exhausts retries melts the breaker once, and a suite that does it repeatedly against a shared service will open a breaker configured with production numbers. ExternalService.reset_all/1 between tests is the other half of the answer.

When you want to keep the real backoff configuration under test — because the delays are the thing you care about — override :sleep_function instead. It is called with each delay in turn, so the test can record them rather than wait for them:

test "backs off exponentially between attempts" do
  ExternalService.start(service,
    circuit_breaker: [tolerate: 100],
    retry: [max_attempts: 4, backoff: :exponential, base: 100],
    sleep_function: recording_sleep()
  )

  ExternalService.call(service, fn -> :retry end)

  assert_slept([100, 200, 400])
end

ExternalService.Test.assert_slept/1 takes the whole sequence rather than one delay at a time, because the sequence is what a backoff configuration determines — a test that checks only the first delay passes against the wrong :backoff.

This is the one place a no-op sleep function is the right tool — the delays are a finite sequence, so skipping them ends the retrying sooner rather than spinning. It is not true of rate limiting, as the warning below explains.

Rate limits

Use wait: false. A throttled call then fails immediately instead of waiting, which is both fast and assertable:

ExternalService.start(service,
  retry: [max_attempts: 1],
  rate_limit: [limit: 1, per: :timer.minutes(1), wait: false]
)

assert ExternalService.call(service, fn -> :first end) == :first

assert {:error, %ExternalService.RateLimited{}} =
         ExternalService.call(service, fn -> flunk("should not run") end)

The alternative is to configure a limit your tests never reach, so the limiter never comes into play at all.

Reaching the failure paths on purpose

You rarely want to test your fallback code by making a real dependency fail. The control API drives the breaker and the limiter directly.

Opening the breaker

ExternalService.Test.trip_breaker/1 opens it without a failing call:

ExternalService.start(service,
  circuit_breaker: [tolerate: 2, within: :timer.seconds(10)],
  retry: [max_attempts: 1]
)

trip_breaker(service)

assert ExternalService.blown?(service)

assert {:error, %ExternalService.CircuitBreakerOpen{}} =
         ExternalService.call(service, fn -> flunk("should not run") end)

Underneath it is ExternalService.CircuitBreaker.melt/1, which reports a failure the library never saw. The reason to use the helper rather than call melt/1 in a loop is an off-by-one: :fuse tolerates :tolerate melts and opens on the next, so tolerate: n needs n + 1 melts. trip_breaker/1 reads :tolerate off the service and does that arithmetic, so the number is not restated in your test and cannot drift from the configuration.

ExternalService.reset/1 closes it again, which is what makes the setup helper above work:

ExternalService.reset(service)
assert ExternalService.available?(service)

Exhausting the rate limit

ExternalService.Test.exhaust_rate_limit/1 puts a service right at its limit before the code under test runs, reading :limit off the service the same way:

ExternalService.start(service,
  retry: [max_attempts: 1],
  rate_limit: [limit: 2, per: :timer.minutes(1), wait: false]
)

exhaust_rate_limit(service)

assert {:error, %ExternalService.RateLimited{}} =
         ExternalService.call(service, fn -> flunk("should not run") end)

It is ExternalService.RateLimiter.request/1 underneath, which spends one call's worth of budget without running anything. Reach for that directly when you want a service partly spent rather than fully.

Degrading a service without touching your code

:fault_injection fails a fraction of calls at the breaker, which is useful for exercising fallback paths under something closer to real conditions. A rate of 1.0 fails every call:

ExternalService.start(service,
  circuit_breaker: [tolerate: 100, within: :timer.seconds(10), fault_injection: 1.0],
  retry: [max_attempts: 1]
)

assert {:error, %ExternalService.CircuitBreakerOpen{}} =
         ExternalService.call(service, fn -> flunk("should not run") end)

Rates between 0.0 and 1.0 are random, so assert on aggregate behavior rather than on any individual call.

Asserting that a retry happened

"Did this actually retry?" is a common thing to want to check, and it is the one question the return value cannot answer: a call that failed twice and then succeeded returns exactly what a call that succeeded first time returns. The telemetry events are the only difference, and ExternalService.Test.record_events/0 puts them where a test can see them:

setup :record_events

test "retries a 503" do
  ExternalService.call(service, fn -> {:retry, :service_unavailable} end)

  assert_retried(service, reason: :service_unavailable)
end

record_events/0 attaches handlers for this library's events, generating a handler ID unique to the calling process and detaching it on exit. That last part matters: handler IDs are global, so a shared one breaks under async: true.

The assertions return the metadata they matched, so you can go further:

metadata = assert_retried(service)
assert metadata.reason == :service_unavailable

refute_retried/2, assert_breaker_blown/2 and assert_throttled/2 cover the other three events. refute_retried/2 earns its place rather than being reflexive symmetry — a retry that did not happen leaves nothing in the return value to assert on instead. All the events are listed in the Telemetry guide.

Making a service inert

Everything above makes a service fast. Sometimes you want it absent: the guarded call should run your function and otherwise stay out of the way, because what you're testing is the code around it.

Child-spec overrides are deep merged with the options given to use ExternalService, so test.exs can neutralize a service without a second module:

children = [
  {MyApp.Stripe,
   circuit_breaker: [tolerate: :infinity],
   rate_limit: [limit: :infinity],
   retry: [max_attempts: 1]}
]

Each key turns off exactly one mechanism, and each is exact rather than merely large:

:infinity matters more than it looks. The breaker and the limiter are the two stateful mechanisms — the ones that accumulate across tests — and until they had an off switch you could only set them very large. Large is finite: a suite long enough to accumulate :tolerate melts starts failing tests that have nothing to do with the breaker, and the failure looks like flakiness. :infinity removes the state rather than postponing it, which also means it no longer matters that a front-door module can't vary its :name per test — inert mechanisms have nothing to share.

Because the merge is deep, you override only the keys you name; :within, :reset, and :per all fall back to the module-level configuration. See Per-environment overrides.

Swap in retry: [max_attempts: 3, base: 0] when you do want the retry logic exercised — base: 0 keeps it instant. And note that rate_limit: [wait: false] is a different thing from limit: :infinity: it makes throttled calls fail fast, which is interference, just quick interference. Use it when you're testing the throttled path, not when you want the limiter gone.

What this library does not give you

There is no single mode: :passthrough switch — the three keys above are the whole of it, deliberately, because each also means something on its own in production. There is also no way to make a guarded call transparent: with max_attempts: 1, a function returning :retry yields RetriesExhausted rather than the raw :retry, because the sentinel belongs to the library.

The broader point: an inert service is not a tested one. Tests that run with the mechanisms off are not exercising your :retry returns, your fallback paths, or your error handling — so keep some that do, using the techniques above. If what you want is for the external call not to happen at all, stub at your own boundary, in the function you pass to call/3, rather than neutralizing the service around it.

Which paths your suite actually exercised

That last point is advice with nothing behind it: nothing tells you whether you took it. A suite can make ten thousand guarded calls, every one on the happy path, and look exactly like a suite that exercises every failure path this library provides.

ExternalService.Test.Coverage answers it, from the telemetry the library already emits:

# test/test_helper.exs
ExUnit.start()
ExternalService.Test.Coverage.install_reporter()

external_service coverage

service             calls    retried     failed    breaker  throttled  saturated
MyApp.Geocoder         44         12          4          0          0          0
MyApp.Search          318          0          0          0          0          0  ⚠
MyApp.Stripe         1204        142         31          3          7          0

⚠ MyApp.Search was called 318 times and never once retried, failed, or was
  rejected. Its `:retry` returns, its fallback path and its error handling are
  not covered by this suite.

Every count is a number of calls, so they are all comparable with the first column — a call that retried four times counts once.

A row of zeros is a prompt, not a verdict. A dependency you stub at your own boundary is supposed to have zeros; that is the recommendation directly above. What the report is for is the service you believed you were testing. It is never a threshold and never a build failure.

The counts can also be read mid-suite, without the reporter, which makes "assert this test exercised the breaker" a thing you can write directly — see ExternalService.Test.Coverage.entries/0.