Which resilience paths your test suite actually exercised.
Testing ends on the point this module gives a mechanism to:
An inert service is not a tested one. Tests that run with the mechanisms off are not exercising your
:retryreturns, your fallback paths, or your error handling.
Nothing tells you whether you took that advice. A suite can make ten thousand guarded calls, every one on the happy path, and look exactly like a suite that exercises every failure path this library exists to provide. Coverage counts the calls per service and how many of them went down each path:
external_service coverage
service calls retried failed breaker throttled saturated
MyApp.Geocoder 44 12 4 0 0 0
MyApp.Search 318 0 0 0 0 0 ⚠
MyApp.Stripe 1204 142 31 3 7 0
⚠ MyApp.Search was called 318 times and never once retried, failed, or was
rejected. Its `:retry` returns, its fallback path and its error handling are
not covered by this suite.A row of zeros is a prompt, not a verdict
A dependency your tests stub at your own boundary is supposed to have zeros — Testing recommends exactly that for business-logic tests. What the report is for is the service you believed you were testing. Either write a test that takes it down a failure path, or know why you did not. This is an opt-in diagnostic you run deliberately, never a threshold and never a build failure.
Enabling
# test/test_helper.exs
ExUnit.start()
ExternalService.Test.Coverage.install_reporter()mix test then prints the table after the suite. Recording needs no reporter,
so entries/0 and report/0 can be read at any point — including mid-suite,
which makes "assert this test exercised the breaker" possible on its own:
attach()
on_exit(&detach/0)
ExternalService.call(service, fn -> {:retry, :nope} end)
assert [%{service: ^service, retried: 1}] = entries()Install the reporter from test_helper.exs
Counts accumulate in an ETS table, and an ETS table dies with the process
that created it. install_reporter/0 creates it from test/test_helper.exs,
whose process outlives the suite. Creating it inside a test discards every
count when that test finishes.
What it costs
Nothing unless attached. Every event it reads is
already emitted, in production too, so there is no
instrumentation to enable and no build that differs — this is a handler and a
reporter. While attached it is one :ets.update_counter/4 per call, plus one
key in the calling process's dictionary for the duration of a call.
Coverage and simulate/3
ExternalService.simulate/3 answers whether a configuration would work.
This answers whether your suite ever found out. Neither substitutes for the
other, and a service with a good simulation and a row of zeros is exactly the
case worth knowing about.
Summary
Types
One service's accumulated coverage. Each count is a number of calls, not of
events, so they are all comparable with :calls — a call that retried four
times counts once.
Types
@type entry() :: %{ service: ExternalService.service(), calls: non_neg_integer(), retried: non_neg_integer(), failed: non_neg_integer(), rejected: non_neg_integer(), throttled: non_neg_integer(), saturated: non_neg_integer(), exercised?: boolean() }
One service's accumulated coverage. Each count is a number of calls, not of
events, so they are all comparable with :calls — a call that retried four
times counts once.
:exercised? is false when a service was called but never once went down any
of the paths, which is the case the report flags.
Functions
@spec attach() :: :ok
Starts recording, creating the counter table if it does not exist.
Called for you by install_reporter/0. Call it directly when you want the
counts without the end-of-suite report — but see the warning in the module
documentation about which process creates the table.
@spec detach() :: :ok
Stops recording. The counts already accumulated survive, and reset/0 clears
them.
@spec entries() :: [entry()]
The accumulated coverage, one entry/0 per service the suite called, ordered
by service.
A service that was never called does not appear — nothing was recorded for it.
@spec install_reporter() :: :ok
Starts recording and prints report/0 after the suite.
Put it in test/test_helper.exs, after ExUnit.start/1.
@spec report() :: String.t()
entries/0 rendered as the report shown in the module documentation.
Returns a message saying so when nothing was recorded, rather than an empty table.
@spec reset() :: :ok
Discards every recorded count. Recording continues if it was attached.