Defines a module-based gateway to an external service.

use ExternalService generates a small, declarative wrapper around the functional API. Configure the circuit breaker, rate limiting, and default retry options at the module level, then start the module under a supervisor and call the service through the generated call/1 (and friends).

Example

defmodule MyApp.Stripe do
  use ExternalService,
    circuit_breaker: [tolerate: 5, within: :timer.seconds(1), reset: :timer.seconds(5)],
    rate_limit: [limit: 100, per: :timer.seconds(1), wait: :timer.seconds(1)],
    retry: [max_attempts: 5, backoff: :exponential, jitter: true]

  def charge(params) do
    call fn ->
      case Stripe.charge(params) do
        {:ok, result} -> {:ok, result}
        {:error, %{status: status}} when status in 500..599 -> :retry
        other -> other
      end
    end
  end
end

Start it under your supervision tree:

children = [MyApp.Stripe]
Supervisor.start_link(children, strategy: :one_for_one)

Configuration can be overridden when starting (useful in tests), and is deep merged with the options given to use:

{MyApp.Stripe, circuit_breaker: [tolerate: 1], retry: [max_attempts: 1]}

Options

Accepts the same options as start/2 (:circuit_breaker, :rate_limit, :retry, :sleep_function), plus:

• :name - the term that identifies the service. Defaults to the module name.

Generated functions

• call/1, call/2, call!/1, call!/2
• call_async/1, call_async/2
• call_async_stream/2, call_async_stream/3, call_async_stream/4
• available?/0, blown?/0, saturated?/0, reset/0, reset_all/0
• child_spec/1, start_link/1

@spec all_available?([service()]) :: boolean()

Returns true only if every service in fuse_names is available?/1.

Useful for guarding work that depends on several services at once.

Examples

if ExternalService.all_available?([:payments, :inventory]) do
  place_order(order)
else
  {:error, :service_unavailable}
end

@spec available?(service()) :: boolean()

Returns true if the service is currently available, meaning its circuit breaker is not blown.

This is useful for the circuit breaker pattern: before kicking off expensive work, you can check whether the services it depends on are available and bail out early (returning a degraded response) if any of them are not.

A service that has not been started (see start/2) is reported as not available. Note that availability can change between this check and a subsequent call/3, so this is a best-effort signal, not a guarantee.

Examples

if ExternalService.available?(:payments) do
  charge(order)
else
  {:error, :payments_unavailable}
end

@spec blown?(service()) :: boolean()

Returns true if the service's circuit breaker is currently blown.

A service that has not been started (see start/2) is not considered blown; use available?/1 if you want "ready to use" semantics that also account for services that were never started.

@spec call(service(), retriable_function()) :: error() | (function_result :: any())

Executes a function for the given service, handling retry and circuit breaker logic.

ExternalService.start/2 must be called for the service before using call.

The provided function can indicate that a retry should be performed by returning the atom :retry or a tuple of the form {:retry, reason}, where reason is any arbitrary term. Any other result is considered successful, so the operation will not be retried and the result of the function will be returned as the result of call.

For functions that were not written to return :retry/{:retry, reason}, the :retry_on retry option takes a predicate that is run on the return value; when it returns a truthy value the call is retried as though the function had returned {:retry, result} (the result becomes the retry reason and the circuit breaker melts). An explicit :retry/{:retry, reason} return always takes precedence over the predicate.

Raised exceptions are only retried if the :retry_exceptions retry option matches them (it defaults to [], matching nothing); otherwise they propagate to the caller untouched. That option takes either a list of exception modules or a predicate run on the exception itself, which can decide per instance rather than per type. An exception that is not retried also does not melt the circuit breaker — :retry_exceptions governs both retrying and whether a raised exception counts as a circuit-breaker failure. When retries are spent on an exception that was being retried, the original exception is re-raised with its original stacktrace.

retry_opts may be a ExternalService.RetryOptions.t/0 struct or a keyword list of retry options. A keyword list is treated as per-call overrides: it is merged onto the service's configured default retry options (from start/2), so it overrides only the keys it lists and inherits the rest. A RetryOptions struct, being a complete set of options, replaces the service defaults entirely. When omitted (the two-argument form call/2), the service's configured defaults are used.

@spec call!(service(), retriable_function()) :: function_result :: any() | no_return()

Like call/3, but raises an exception if retries are exhausted or the circuit breaker is open.

@spec call_async(service(), retriable_function()) :: Task.t()

Asynchronous version of ExternalService.call.

Returns a Task that may be used to retrieve the result of the async call.

@spec call_async_stream(Enumerable.t(), service(), (any() ->
                                                retriable_function_result())) ::
  Enumerable.t()

Parallel, streaming version of ExternalService.call.

See call_async_stream/5 for full documentation.

@spec call_async_stream(
  Enumerable.t(),
  service(),
  ExternalService.RetryOptions.t() | (async_opts :: list()),
  (any() -> retriable_function_result())
) :: Enumerable.t()

Parallel, streaming version of ExternalService.call.

See call_async_stream/5 for full documentation.

@spec call_async_stream(
  Enumerable.t(),
  service(),
  ExternalService.RetryOptions.t() | keyword() | nil,
  async_opts :: list(),
  (any() -> retriable_function_result())
) :: Enumerable.t()

Parallel, streaming version of ExternalService.call.

This function uses Elixir's built-in Task.async_stream/3 function and the description below is taken from there.

Returns a stream that runs the given function function concurrently on each item in enumerable.

Each enumerable item is passed as argument to the given function function and processed by its own task. The tasks will be linked to the current process, similarly to async/1.

@spec explain(service() | keyword()) :: String.t()

Describes what a configuration will do, as a report meant to be read.

IO.puts ExternalService.explain(MyApp.Stripe)

Takes either a started service or a keyword list of options, so a configuration can be examined before it ships as well as after:

IO.puts ExternalService.explain(
  circuit_breaker: [tolerate: 3],
  retry: [base: 100, max_attempts: 5]
)

Everything in the report is derived from the options rather than measured, and the checks described above are included in it, so this is also the way to see why a service warned at compile time.

Example

:payments

  retry
    window       1.5s
    delays       100ms, 200ms, 400ms, 800ms
    attempts     up to 5
    time budget  none (:expiry unset)

  circuit breaker
    opens after      4 failing calls
    counting window  10.0s
    resets after     60.0s
    backend          ExternalService.CircuitBreaker.Fuse

  rate limit
    none  calls are not throttled

  concurrency
    none  calls are not limited in flight

  a fully-failing call
    spends  1.5s waiting between attempts
    plus    however long its 5 attempts take — nothing here bounds a single attempt

Note that a keyword list is always read as options. A service identified by a list has to be explained through the started-service path, which means starting it first.

@spec rate_limited?(service()) :: boolean()

Returns true if a call to the service would currently be throttled by its rate limit.

This is a read: it consumes none of the service's budget, so it is safe to ask speculatively before committing to expensive work. A service with no rate limit configured is never rate limited.

As with available?/1, this is a best-effort signal — the answer can change between the check and a subsequent call/3. Use ExternalService.RateLimiter.peek/1 when you want to know how long the wait would be rather than merely whether there is one.

Examples

if ExternalService.rate_limited?(:payments) do
  {:error, :busy}
else
  charge(order)
end

@spec reset(service()) :: :ok | {:error, :not_found}

Resets the circuit breaker for the given service.

After reset, the breaker will be closed with no recorded failures.

@spec reset_all(service()) :: :ok | {:error, :not_found}

Resets every stateful mechanism for the given service: the circuit breaker, the rate limiter, and the concurrency limit.

reset/1 closes the breaker only, and deliberately leaves the rate limiter alone — clearing a limiter in production releases a burst at the service, which is rarely what someone closing a breaker meant to do. This function is for when you do want a clean slate.

Its main use is between tests. A service's state is global — it lives in :persistent_term and :fuse, keyed on the service term — so a test that trips the breaker or drains the rate limit budget leaves it that way for whatever runs next:

setup do
  ExternalService.reset_all(MyApp.Stripe)
  :ok
end

A service with no rate limit configured resets just the breaker. One that was never started answers {:error, :not_found}, exactly as reset/1 does.

Note that this shares state rather than isolating it, so it does not make concurrent tests independent — see the Testing guide.

@spec saturated?(service()) :: boolean()

Returns true if service has no concurrency slot free right now.

Completes the trio with available?/1 (is the breaker closed?) and rate_limited?/1 (would a call be throttled?). A service with no :concurrency limit configured — including one that was never started — is never saturated, since nothing is holding calls back.

Like the others this is a best-effort signal: a slot can be taken or released between the check and a subsequent call, so it lets you bail out early rather than replacing handling of ExternalService.ServiceSaturated from the call itself.

@spec simulate(service() | keyword(), scenario(), keyword()) ::
  ExternalService.Simulation.t()

Runs a configuration against a failing dependency and reports what happened.

explain/1 says what a configuration is; this says what it does. The question it answers is the one every resilience configuration has and few are ever asked: does the breaker actually open, how long does a failing call take, and how much load does a dead dependency absorb first?

test "our breaker actually opens, and fast enough" do
  assert %ExternalService.Simulation{opens_after: opens, worst_call: worst} =
           ExternalService.simulate(MyApp.Stripe, :always_failing)

  assert opens <= 5
  assert worst < 2_000
end

Takes a started service or a keyword list of options, like explain/1.

Scenarios

• :always_failing — every attempt fails, instantly. The base case.
• {:always_failing, attempt_ms} — every attempt fails and takes attempt_ms. Attempt duration is the one thing a configuration cannot state, and the thing that makes a hand-sized :within too narrow, so this is the scenario worth running against a dependency you know to be slow.
• {:slow, attempt_ms} — attempts succeed but take attempt_ms each. Nothing fails, so the breaker never opens; :worst_call is the answer here.
• {:failing_for, ms} — fails for the first ms of simulated time, then recovers.
• {:intermittent, rate} — each attempt fails with probability rate. Seed :rand for a reproducible run.

Options

• :max_calls — how many calls to simulate before giving up on the breaker opening. Defaults to 100.

What it does and does not model

It runs on a virtual clock, so simulating half an hour of a background job costs microseconds and no test waits for anything. Delays are nominal, with :jitter switched off — the same choice RetryOptions.window/1 and explain/1 make, so that all three agree and a simulation asserted in a test does not vary between runs. Jitter changes the exact :worst_call, never whether a breaker opens.

The delays come from the library's own planner, the options are resolved through the same path start/2 uses, and melting follows the service's :melt setting. The one thing modelled rather than executed is the circuit breaker's sliding failure window, which is what makes a virtual clock possible at all — a real breaker counts against real time. That model is pinned against measured behavior in the test suite, including a configuration that stays closed through twelve consecutive failing calls.

The rate limiter and the concurrency limit are not simulated. Neither melts the breaker, and both govern the traffic reaching a service rather than what the service does with a failure.

Simulated calls arrive one after another, which is the worst case for opening a breaker: concurrent callers deliver failures closer together, so a breaker that opens here opens at least as readily under real traffic.

Examples

iex> %ExternalService.Simulation{opens_after: opens} =
...>   ExternalService.simulate(
...>     [circuit_breaker: [tolerate: 3], retry: [base: 100, max_attempts: 5]],
...>     :always_failing
...>   )
iex> opens
4

@spec start(service(), options()) :: :ok

Initializes the circuit breaker (and optional rate limiting and default retry options) for a specific service.

The service is a term that uniquely identifies an external service within the scope of an application.

Configuration checks

These options are validated individually, and then checked against each other — because the mistakes worth catching are pairs of options that are each valid and jointly wrong. A window narrower than the failures it has to count, or ten attempts of uncapped exponential backoff, produce a warning naming the setting to change and a value to try.

Services declared with use ExternalService are checked at compile time, so the warning carries a file and a line and fails a build compiled with --warnings-as-errors. Services started through this function are checked here, which is also what covers child-spec overrides, since those are runtime values.

Combinations that cannot work at all — rather than merely being unwise — raise instead. Set how findings are reported with:

config :external_service, on_suspicious_config: :warn   # | :raise | :ignore

:warn is the default. :raise is worth setting in a test environment, where a suspicious configuration is better as a failure than as a log line. :ignore is for a configuration you have decided is right despite what the checks make of it.

Options

• :circuit_breaker (keyword/0) - Circuit breaker configuration. The default value is [].

  • :tolerate - Number of failures tolerated within the :within window before the breaker opens. What counts as one failure is set by :melt, and defaults to one failing call — so tolerate: 3 means three dead calls, whatever :max_attempts is. :infinity installs no breaker at all: it never opens, holds no state, and cannot be combined with :fault_injection. The default value is 10.

  • :melt - What one unit of :tolerate counts. :per_call (the default) melts the breaker once per call, when its retrying gives up, so :tolerate is denominated in calls and is independent of :max_attempts. :per_attempt melts on every failing attempt, which is how versions before 3.0 behaved: a single call with max_attempts: 5 then contributes up to 5, and :tolerate cannot be tuned independently of the retry options. :per_call requires retrying to be bounded — see the note on unbounded retries below. The default value is :per_call.

  • :within - Length of the failure-counting window, in milliseconds. :auto (the default) sizes the window against the retry options instead, since how long it takes :tolerate failures to arrive depends on how long a failing call takes. It is a floor, never narrower than the 10 seconds it replaces — see Sizing the window for when to set it yourself. The default value is :auto.

  • :reset (pos_integer/0) - Milliseconds to wait before the breaker resets (closes) after it has opened. The default value is 60000.

  • :fault_injection (float/0) - If set to a rate between 0.0 and 1.0, randomly fails that fraction of calls. Intended for testing how dependents behave when this service is degraded.

  • :backend - The circuit breaker implementation to use, either as a module or as a {module, options} tuple whose options are passed through to that backend. Defaults to the node-local :fuse-based breaker.

• :rate_limit (keyword/0) - Optional rate-limiting configuration. Omit for no rate limiting.

  • :limit - Required. Maximum number of calls allowed within each :per window. :infinity installs no limiter at all — calls pass straight through, exactly as if :rate_limit had been omitted. It is meant for overriding a configured limit (a child spec override cannot remove a key); to have no rate limiting in the first place, omit :rate_limit. :per is still required, and ignored.

  • :per (pos_integer/0) - Required. Length of the rate-limiting window, in milliseconds.

  • :wait - How long a throttled call may wait for the rate limit to admit it. :infinity waits as long as it takes, false never waits, and an integer is a millisecond budget for the whole call. When the budget runs out the call is not made and an ExternalService.RateLimited error is returned (or raised by call!/3). Defaults to one window — :per, capped at 5 seconds — which absorbs a burst without converting sustained throttling into unbounded latency. Use :infinity for background work and pipelines, where sleeping is how back-pressure propagates upstream.

  • :backend - The rate limiter implementation to use, either as a module or as a {module, options} tuple whose options are passed through to that backend. Defaults to ExternalService.RateLimiter.Local. Use ExternalService.RateLimiter.Hammer for a limit shared across a cluster.

• :concurrency (keyword/0) - Optional concurrency limit (the bulkhead pattern). Omit for no limit. See ExternalService.Concurrency.

  • :limit (pos_integer/0) - Required. Maximum number of calls allowed to be in flight against the service at once.

  • :reclaim_after (pos_integer/0) - Required. Milliseconds after which a held slot is considered abandoned and may be reused. A slot is normally released as soon as the call finishes, but a caller killed from outside — including by the ordinary :shutdown a supervisor sends while draining — never runs its release. This bounds how long such a slot is lost. Required rather than defaulted: it must exceed the longest legitimate call, which depends on the timeout configured in your HTTP client. Setting it too low silently admits more than :limit.

  • :wait - How long a call may wait for a slot before being shed. false (the default) sheds immediately; an integer is a millisecond budget. A short budget absorbs bursts without allowing a pile-up, since waiting callers hold no slot and are bounded by arrival rate times the budget. Unlike the rate limiter's :wait, :infinity is not accepted — an unbounded wait is the pile-up a concurrency limit exists to prevent. The default value is false.

• :retry - Default retry options for the service, used by call/2. See ExternalService.RetryOptions for the available keys. The default value is [].

• :sleep_function (function of arity 1) - Overrides the function used whenever a call waits — between retry attempts, while rate limited, and while waiting for a concurrency slot. Called with the number of milliseconds to wait. Defaults to Process.sleep/1; overriding it lets tests exercise backoff and throttling without waiting for them.

@spec stop(service()) :: :ok

Stops the fuse for a specific service.

Stopping is idempotent: it is safe to call on a service that was never started or has already been stopped.