ExternalService.Test (ExternalService v3.1.0)

ExUnit helpers for the setup and assertions the Testing guide otherwise asks you to hand-write.

Each one replaces a passage of that guide where the work is repeated verbatim, is arithmetic the library can do from your own configuration, or is easy to get wrong in a way that fails silently:

Helper	Replaces
`trip_breaker/1`	melting `:tolerate` + 1 times, with the off-by-one written out
`exhaust_rate_limit/1`	spending `:limit` calls' worth of budget by hand
`record_events/0` and the assertions	twelve lines of `:telemetry.attach` per test
`recording_sleep/1` and `assert_slept/1`	a hand-rolled `:sleep_function` shim

The guide still explains why each of these is the technique to use. These replace the typing, not the model.

Usage

defmodule MyApp.StripeTest do
  use ExUnit.Case, async: true
  use ExternalService.Test

  setup :record_events

  setup do
    ExternalService.start(MyApp.Stripe,
      circuit_breaker: [tolerate: 2, within: :timer.seconds(10)],
      retry: [max_attempts: 3, base: 0]
    )

    on_exit(fn -> ExternalService.stop(MyApp.Stripe) end)
  end

  test "retries a 503, then gives up" do
    result = ExternalService.call(MyApp.Stripe, fn -> {:retry, :service_unavailable} end)

    assert {:error, %ExternalService.RetriesExhausted{}} = result
    assert_retried(MyApp.Stripe, reason: :service_unavailable)
  end

  test "fails fast once the breaker is open" do
    trip_breaker(MyApp.Stripe)

    assert {:error, %ExternalService.CircuitBreakerOpen{}} =
             ExternalService.call(MyApp.Stripe, fn -> flunk("should not run") end)

    assert_breaker_blown(MyApp.Stripe)
  end
end

use ExternalService.Test is exactly import ExternalService.Test.

Matching rules

The assertions take an optional keyword of expectations checked against the event's telemetry metadata. An expectation is met when the metadata field is equal to it, or — for a Regex — when the field's string form matches. A field named in the expectations but absent from the metadata never matches.

Each assertion returns the metadata map it matched, so further assertions can be made on it:

metadata = assert_retried(MyApp.Stripe)
assert metadata.reason == :service_unavailable

Service state is global

These helpers do not change that, and none of them isolates one test from another. trip_breaker/1 and exhaust_rate_limit/1 in particular leave the service tripped and spent for whatever runs next against the same service term — see Isolating tests from each other.

Summary

Functions

__using__(opts)

Convenience: use ExternalService.Test is equivalent to import ExternalService.Test.

assert_breaker_blown(service, expectations \\ [])

Asserts that a call to service was rejected by an open circuit breaker.

assert_retried(service, expectations \\ [])

Asserts that service retried at least once.

assert_slept(delays)

Asserts that the delays recorded by recording_sleep/1 were exactly delays, in order.

assert_throttled(service, expectations \\ [])

Asserts that a call to service was throttled and put to sleep.

exhaust_rate_limit(service)

Spends the whole rate-limit budget for service, without running anything.

record_events()

Records this library's telemetry events to the calling process for the rest of the test.

record_events(context)

Same as record_events/0, ignoring the ExUnit context.

recording_sleep(pid \\ self())

Builds a :sleep_function that records each delay to pid instead of waiting.

refute_retried(service, expectations \\ [])

Asserts that service did not retry.

trip_breaker(service)

Opens the circuit breaker for service, without needing a failing call.

Functions

using(opts)

(macro)

Convenience: use ExternalService.Test is equivalent to import ExternalService.Test.

assert_breaker_blown(service, expectations \\ [])

@spec assert_breaker_blown(
  ExternalService.service(),
  keyword()
) :: map()

Asserts that a call to service was rejected by an open circuit breaker.

assert_breaker_blown(service)

Requires record_events/0. Returns the metadata of the matching event.

Note that this asserts a call was rejected, which is not the same as the breaker being open: use ExternalService.blown?/1 for the state itself.

assert_retried(service, expectations \\ [])

@spec assert_retried(
  ExternalService.service(),
  keyword()
) :: map()

Asserts that service retried at least once.

A retry is invisible in a call's return value — a call that failed twice and then succeeded returns exactly what a call that succeeded first time returns — so this is the only way to see one.

assert_retried(service)
assert_retried(service, reason: :service_unavailable)

Requires record_events/0. Returns the metadata of the matching event.

assert_slept(delays)

@spec assert_slept([non_neg_integer()]) :: [non_neg_integer()]

Asserts that the delays recorded by recording_sleep/1 were exactly delays, in order.

assert_slept([100, 200, 400])

Asserts on the whole sequence rather than on each delay separately, because the sequence is the thing a backoff configuration determines — a test that checks only the first delay passes against the wrong :backoff. Pass [] to assert that nothing slept at all.

Returns the delays.

assert_throttled(service, expectations \\ [])

@spec assert_throttled(
  ExternalService.service(),
  keyword()
) :: map()

Asserts that a call to service was throttled and put to sleep.

assert_throttled(service)

Requires record_events/0. Returns the metadata of the matching event.

Only fires when the call actually waits, so a service configured with wait: false throttles by returning ExternalService.RateLimited and this never matches. Assert on that error instead.

exhaust_rate_limit(service)

@spec exhaust_rate_limit(ExternalService.service()) :: ExternalService.service()

Spends the whole rate-limit budget for service, without running anything.

Reads :limit off the service and requests that many times, so the next call is throttled. Returns the service.

exhaust_rate_limit(service)

assert {:error, %ExternalService.RateLimited{}} =
         ExternalService.call(service, fn -> flunk("should not run") end)

Pair it with wait: false, or the throttled call waits for the window rather than failing — see Rate limits.

Raises if the service was never started, or was started with no rate limit.

record_events()

@spec record_events() :: :ok

Records this library's telemetry events to the calling process for the rest of the test.

Intended for a setup block, though it works anywhere before the code under test runs:

setup :record_events

Handler IDs are global, so this generates one unique to the calling process and detaches it with ExUnit.Callbacks.on_exit/2 — which is what makes the assertions safe under async: true.

Returns :ok, so it satisfies setup's contract directly.

record_events(context)

@spec record_events(term()) :: :ok

Same as record_events/0, ignoring the ExUnit context.

This is the arity setup :record_events calls; record_events/0 is the one to call from a test body or from a setup do block.

recording_sleep(pid \\ self())

@spec recording_sleep(pid()) :: (non_neg_integer() -> :ok)

Builds a :sleep_function that records each delay to pid instead of waiting.

ExternalService.start(service,
  retry: [max_attempts: 4, backoff: :exponential, base: 100],
  sleep_function: recording_sleep()
)

ExternalService.call(service, fn -> :retry end)

assert_slept([100, 200, 400])

This keeps the real backoff configuration under test while taking the waiting off the clock, which a base: 0 override cannot do.

Retry backoff only

A :sleep_function that does not sleep is right for retry backoff, whose delays are a fixed sequence, and wrong for rate limiting and concurrency, which re-check a condition in a loop. There, not sleeping does not skip the wait — the limiter is asked again immediately, still says wait, and the loop spins until real time has passed. Measured at limit: 1, per: 2_000, the throttled call still took 2000ms and invoked the function 2,075,418 times.

Use wait: false for those. See Keeping tests off the clock.

refute_retried(service, expectations \\ [])

@spec refute_retried(
  ExternalService.service(),
  keyword()
) :: :ok

Asserts that service did not retry.

Unlike most refutations this is not reflexive symmetry: a retry that did not happen leaves nothing in the return value to assert on instead.

refute_retried(service)

Requires record_events/0.

trip_breaker(service)

@spec trip_breaker(ExternalService.service()) :: ExternalService.service()

Opens the circuit breaker for service, without needing a failing call.

:fuse tolerates :tolerate melts and opens on the next, so this melts :tolerate + 1 times — reading the number off the service rather than asking the test to restate it. Melting is direct, so melt: :per_attempt makes no difference to the count.

Returns the service, so it composes in a setup.

setup context do
  ExternalService.start(context.service, circuit_breaker: [tolerate: 2])
  trip_breaker(context.service)
  :ok
end

Raises if the service was never started, or if it was started with tolerate: :infinity, which installs no breaker at all and therefore has nothing to open.