# `ExternalService`
[🔗](https://github.com/jvoegele/external_service/blob/main/lib/external_service.ex#L1)

ExternalService handles all retry and circuit breaker logic for calls to external services.

The recommended way to use it is the declarative module-based front door,
`use ExternalService` (see `__using__/1`), which lets you configure a service's
circuit breaker, rate limiting, and default retry options in one place. The
functional API (`start/2`, `call/3`, and friends) is the lower-level foundation
it is built on, and can be used directly when you need more control.

## Telemetry

`ExternalService` emits [`:telemetry`](https://hexdocs.pm/telemetry) events so
that calls to external services can be observed and instrumented. Attach a
handler to any of the events below to forward them to your metrics or logging
backend.

All events carry a `:service` key in their metadata, which is the name of the
service the event relates to.

  * `[:external_service, :call, :start]` - emitted when a guarded call begins.
    * Measurements: `:system_time`, `:monotonic_time`
    * Metadata: `:service`

  * `[:external_service, :call, :stop]` - emitted when a guarded call completes
    (including when it returns an error such as `ExternalService.RetriesExhausted`
    or `ExternalService.CircuitBreakerOpen`).
    * Measurements: `:duration`, `:monotonic_time`
    * Metadata: `:service`, `:result` (the value returned from the call)

  * `[:external_service, :call, :exception]` - emitted when a guarded call
    raises (for example a non-retriable exception, or `call!/3` raising on an
    open circuit breaker or exhausted retries).
    * Measurements: `:duration`, `:monotonic_time`
    * Metadata: `:service`, `:kind`, `:reason`, `:stacktrace`

  * `[:external_service, :call, :retry]` - emitted each time a call's function
    fails retriably: it returned `:retry` / `{:retry, reason}`, it returned a
    result matched by the `:retry_on` predicate, or it raised an exception matched
    by the `:retry_exceptions` retry option. Exceptions `:retry_exceptions` does
    not match neither count as a failure nor emit this event. Whether another
    attempt is actually made depends on the retry options.

    This counts **attempts**, under either `:melt` setting — a failed attempt is
    worth observing whether or not it charges the circuit breaker. It is
    therefore not a melt count: with the default `melt: :per_call`, a call that
    retries four times and then succeeds emits this four times and melts nothing.
    * Measurements: `:count` (always `1`)
    * Metadata: `:service`, `:reason`

  * `[:external_service, :circuit_breaker, :blown]` - emitted when a call is
    rejected because the service's circuit breaker is blown.
    * Measurements: `:count` (always `1`)
    * Metadata: `:service`

  * `[:external_service, :rate_limit, :sleep]` - emitted when a call is
    throttled and put to sleep to stay within the configured rate limit.
    * Measurements: `:sleep_time` (milliseconds)
    * Metadata: `:service`

# `circuit_breaker_open`

```elixir
@type circuit_breaker_open() :: {:error, ExternalService.CircuitBreakerOpen.t()}
```

Error returned when a service's circuit breaker is open

# `error`

```elixir
@type error() ::
  retries_exhausted()
  | circuit_breaker_open()
  | service_not_started()
  | rate_limited()
```

Union type representing all the possible error return values

# `melt`

```elixir
@type melt() :: :per_call | :per_attempt
```

What one unit of the circuit breaker's `:tolerate` counts: a failing call, or a
failing attempt. See the `:melt` circuit breaker option under `start/2`.

# `options`

```elixir
@type options() :: keyword()
```

Options for `start/2`. See the schema documented under `start/2`.

# `rate_limited`

```elixir
@type rate_limited() :: {:error, ExternalService.RateLimited.t()}
```

Error returned when a call is throttled beyond the rate limit `:wait` budget

# `retriable_function`

```elixir
@type retriable_function() :: (-> retriable_function_result())
```

# `retriable_function_result`

```elixir
@type retriable_function_result() ::
  :retry | {:retry, reason :: any()} | (function_result :: any())
```

# `retries_exhausted`

```elixir
@type retries_exhausted() :: {:error, ExternalService.RetriesExhausted.t()}
```

Error returned when the allowable number of retries has been exceeded

# `scenario`

```elixir
@type scenario() ::
  :always_failing
  | {:always_failing, attempt_ms :: non_neg_integer()}
  | {:slow, attempt_ms :: non_neg_integer()}
  | {:failing_for, ms :: non_neg_integer()}
  | {:intermittent, rate :: float()}
```

A dependency to simulate a configuration against. See `simulate/3`.

# `service`

```elixir
@type service() :: term()
```

A term that uniquely identifies an external service.

# `service_not_started`

```elixir
@type service_not_started() :: {:error, ExternalService.ServiceNotStarted.t()}
```

Error returned when a service has not been started with `ExternalService.start/2`

# `sleep_function`

```elixir
@type sleep_function() :: (non_neg_integer() -> any())
```

The function used whenever a call waits: between retry attempts, while throttled
to stay within the rate limit, and while waiting for a concurrency slot.

Blocking the calling process for an extended period is sometimes undesirable
(for example in tests, where it is the difference between a suite that waits out
every backoff and one that does not), so this can be overridden. It is called
with the number of milliseconds to wait. Defaults to `Process.sleep/1`.

# `__using__`
*macro* 

Defines a module-based gateway to an external service.

`use ExternalService` generates a small, declarative wrapper around the
functional API. Configure the circuit breaker, rate limiting, and default
retry options at the module level, then start the module under a supervisor
and call the service through the generated `call/1` (and friends).

## Example

    defmodule MyApp.Stripe do
      use ExternalService,
        circuit_breaker: [tolerate: 5, within: :timer.seconds(1), reset: :timer.seconds(5)],
        rate_limit: [limit: 100, per: :timer.seconds(1), wait: :timer.seconds(1)],
        retry: [max_attempts: 5, backoff: :exponential, jitter: true]

      def charge(params) do
        call fn ->
          case Stripe.charge(params) do
            {:ok, result} -> {:ok, result}
            {:error, %{status: status}} when status in 500..599 -> :retry
            other -> other
          end
        end
      end
    end

Start it under your supervision tree:

    children = [MyApp.Stripe]
    Supervisor.start_link(children, strategy: :one_for_one)

Configuration can be overridden when starting (useful in tests), and is deep
merged with the options given to `use`:

    {MyApp.Stripe, circuit_breaker: [tolerate: 1], retry: [max_attempts: 1]}

## Options

Accepts the same options as `start/2` (`:circuit_breaker`, `:rate_limit`,
`:retry`, `:sleep_function`), plus:

  * `:name` - the term that identifies the service. Defaults to the module name.

## Generated functions

  * `call/1`, `call/2`, `call!/1`, `call!/2`
  * `call_async/1`, `call_async/2`
  * `call_async_stream/2`, `call_async_stream/3`, `call_async_stream/4`
  * `available?/0`, `blown?/0`, `saturated?/0`, `reset/0`, `reset_all/0`
  * `child_spec/1`, `start_link/1`

# `all_available?`

```elixir
@spec all_available?([service()]) :: boolean()
```

Returns `true` only if every service in `fuse_names` is `available?/1`.

Useful for guarding work that depends on several services at once.

## Examples

    if ExternalService.all_available?([:payments, :inventory]) do
      place_order(order)
    else
      {:error, :service_unavailable}
    end

# `available?`

```elixir
@spec available?(service()) :: boolean()
```

Returns `true` if the service is currently available, meaning its circuit
breaker is not blown.

This is useful for the circuit breaker pattern: before kicking off expensive
work, you can check whether the services it depends on are available and bail
out early (returning a degraded response) if any of them are not.

A service that has not been started (see `start/2`) is reported as not
available. Note that availability can change between this check and a
subsequent `call/3`, so this is a best-effort signal, not a guarantee.

## Examples

    if ExternalService.available?(:payments) do
      charge(order)
    else
      {:error, :payments_unavailable}
    end

# `blown?`

```elixir
@spec blown?(service()) :: boolean()
```

Returns `true` if the service's circuit breaker is currently blown.

A service that has not been started (see `start/2`) is _not_ considered blown;
use `available?/1` if you want "ready to use" semantics that also account for
services that were never started.

# `call`

```elixir
@spec call(service(), retriable_function()) :: error() | (function_result :: any())
```

Executes a function for the given service, handling retry and circuit breaker logic.

`ExternalService.start/2` must be called for the service before using `call`.

The provided function can indicate that a retry should be performed by returning the atom
`:retry` or a tuple of the form `{:retry, reason}`, where `reason` is any arbitrary term. Any
other result is considered successful, so the operation will not be retried and the result of
the function will be returned as the result of `call`.

For functions that were not written to return `:retry`/`{:retry, reason}`, the `:retry_on` retry
option takes a predicate that is run on the return value; when it returns a truthy value the call
is retried as though the function had returned `{:retry, result}` (the result becomes the retry
reason and the circuit breaker melts). An explicit `:retry`/`{:retry, reason}` return always
takes precedence over the predicate.

Raised exceptions are only retried if the `:retry_exceptions` retry option matches them (it
defaults to `[]`, matching nothing); otherwise they propagate to the caller untouched. That
option takes either a list of exception modules or a predicate run on the exception itself, which
can decide per *instance* rather than per type. An exception that is not retried also does *not*
melt the circuit breaker — `:retry_exceptions` governs both retrying and whether a raised
exception counts as a circuit-breaker failure. When retries are spent on an exception that *was*
being retried, the original exception is re-raised with its original stacktrace.

`retry_opts` may be a `t:ExternalService.RetryOptions.t/0` struct or a keyword list of retry
options. A keyword list is treated as per-call *overrides*: it is merged onto the service's
configured default retry options (from `start/2`), so it overrides only the keys it lists and
inherits the rest. A `RetryOptions` struct, being a complete set of options, replaces the
service defaults entirely. When omitted (the two-argument form `call/2`), the service's
configured defaults are used.

# `call`

```elixir
@spec call(
  service(),
  ExternalService.RetryOptions.t() | keyword(),
  retriable_function()
) ::
  error() | (function_result :: any())
```

# `call!`

```elixir
@spec call!(service(), retriable_function()) :: function_result :: any() | no_return()
```

Like `call/3`, but raises an exception if retries are exhausted or the circuit breaker is open.

# `call!`

```elixir
@spec call!(
  service(),
  ExternalService.RetryOptions.t() | keyword(),
  retriable_function()
) ::
  function_result :: any() | no_return()
```

# `call_async`

```elixir
@spec call_async(service(), retriable_function()) :: Task.t()
```

Asynchronous version of `ExternalService.call`.

Returns a `Task` that may be used to retrieve the result of the async call.

# `call_async`

```elixir
@spec call_async(
  service(),
  ExternalService.RetryOptions.t() | keyword(),
  retriable_function()
) ::
  Task.t()
```

# `call_async_stream`

```elixir
@spec call_async_stream(Enumerable.t(), service(), (any() ->
                                                retriable_function_result())) ::
  Enumerable.t()
```

Parallel, streaming version of `ExternalService.call`.

See `call_async_stream/5` for full documentation.

# `call_async_stream`

```elixir
@spec call_async_stream(
  Enumerable.t(),
  service(),
  ExternalService.RetryOptions.t() | (async_opts :: list()),
  (any() -> retriable_function_result())
) :: Enumerable.t()
```

Parallel, streaming version of `ExternalService.call`.

See `call_async_stream/5` for full documentation.

# `call_async_stream`

```elixir
@spec call_async_stream(
  Enumerable.t(),
  service(),
  ExternalService.RetryOptions.t() | keyword() | nil,
  async_opts :: list(),
  (any() -> retriable_function_result())
) :: Enumerable.t()
```

Parallel, streaming version of `ExternalService.call`.

This function uses Elixir's built-in `Task.async_stream/3` function and the description below is
taken from there.

Returns a stream that runs the given function `function` concurrently on each
item in `enumerable`.

Each `enumerable` item is passed as argument to the given function `function`
and processed by its own task. The tasks will be linked to the current
process, similarly to `async/1`.

# `explain`

```elixir
@spec explain(service() | keyword()) :: String.t()
```

Describes what a configuration will do, as a report meant to be read.

    IO.puts ExternalService.explain(MyApp.Stripe)

Takes either a started service or a keyword list of options, so a configuration
can be examined before it ships as well as after:

    IO.puts ExternalService.explain(
      circuit_breaker: [tolerate: 3],
      retry: [base: 100, max_attempts: 5]
    )

Everything in the report is derived from the options rather than measured, and
the checks described above are included in it, so this is also the way to see why
a service warned at compile time.

## Example

    :payments

      retry
        window       1.5s
        delays       100ms, 200ms, 400ms, 800ms
        attempts     up to 5
        time budget  none (:expiry unset)

      circuit breaker
        opens after      4 failing calls
        counting window  10.0s
        resets after     60.0s
        backend          ExternalService.CircuitBreaker.Fuse

      rate limit
        none  calls are not throttled

      concurrency
        none  calls are not limited in flight

      a fully-failing call
        spends  1.5s waiting between attempts
        plus    however long its 5 attempts take — nothing here bounds a single attempt

Note that a keyword list is always read as options. A service identified by a
list has to be explained through the started-service path, which means starting
it first.

# `rate_limited?`

```elixir
@spec rate_limited?(service()) :: boolean()
```

Returns `true` if a call to the service would currently be throttled by its
rate limit.

This is a *read*: it consumes none of the service's budget, so it is safe to
ask speculatively before committing to expensive work. A service with no rate
limit configured is never rate limited.

As with `available?/1`, this is a best-effort signal — the answer can change
between the check and a subsequent `call/3`. Use
`ExternalService.RateLimiter.peek/1` when you want to know *how long* the wait
would be rather than merely whether there is one.

## Examples

    if ExternalService.rate_limited?(:payments) do
      {:error, :busy}
    else
      charge(order)
    end

# `reset`

```elixir
@spec reset(service()) :: :ok | {:error, :not_found}
```

Resets the circuit breaker for the given service.

After reset, the breaker will be closed with no recorded failures.

# `reset_all`

```elixir
@spec reset_all(service()) :: :ok | {:error, :not_found}
```

Resets every stateful mechanism for the given service: the circuit breaker, the
rate limiter, and the concurrency limit.

`reset/1` closes the breaker only, and deliberately leaves the rate limiter
alone — clearing a limiter in production releases a burst at the service, which
is rarely what someone closing a breaker meant to do. This function is for when
you do want a clean slate.

Its main use is between tests. A service's state is global — it lives in
`:persistent_term` and `:fuse`, keyed on the service term — so a test that
trips the breaker or drains the rate limit budget leaves it that way for
whatever runs next:

    setup do
      ExternalService.reset_all(MyApp.Stripe)
      :ok
    end

A service with no rate limit configured resets just the breaker. One that was
never started answers `{:error, :not_found}`, exactly as `reset/1` does.

Note that this shares state rather than isolating it, so it does not make
concurrent tests independent — see the [Testing](testing.md) guide.

# `saturated?`

```elixir
@spec saturated?(service()) :: boolean()
```

Returns `true` if `service` has no concurrency slot free right now.

Completes the trio with `available?/1` (is the breaker closed?) and
`rate_limited?/1` (would a call be throttled?). A service with no
`:concurrency` limit configured — including one that was never started — is
never saturated, since nothing is holding calls back.

Like the others this is a best-effort signal: a slot can be taken or released
between the check and a subsequent call, so it lets you bail out early rather
than replacing handling of `ExternalService.ServiceSaturated` from the call
itself.

# `simulate`

```elixir
@spec simulate(service() | keyword(), scenario(), keyword()) ::
  ExternalService.Simulation.t()
```

Runs a configuration against a failing dependency and reports what happened.

`explain/1` says what a configuration *is*; this says what it *does*. The
question it answers is the one every resilience configuration has and few are
ever asked: does the breaker actually open, how long does a failing call take,
and how much load does a dead dependency absorb first?

    test "our breaker actually opens, and fast enough" do
      assert %ExternalService.Simulation{opens_after: opens, worst_call: worst} =
               ExternalService.simulate(MyApp.Stripe, :always_failing)

      assert opens <= 5
      assert worst < 2_000
    end

Takes a started service or a keyword list of options, like `explain/1`.

## Scenarios

  * `:always_failing` — every attempt fails, instantly. The base case.
  * `{:always_failing, attempt_ms}` — every attempt fails and takes `attempt_ms`.
    Attempt duration is the one thing a configuration cannot state, and the thing
    that makes a hand-sized `:within` too narrow, so this is the scenario worth
    running against a dependency you know to be slow.
  * `{:slow, attempt_ms}` — attempts succeed but take `attempt_ms` each. Nothing
    fails, so the breaker never opens; `:worst_call` is the answer here.
  * `{:failing_for, ms}` — fails for the first `ms` of simulated time, then
    recovers.
  * `{:intermittent, rate}` — each attempt fails with probability `rate`. Seed
    `:rand` for a reproducible run.

## Options

  * `:max_calls` — how many calls to simulate before giving up on the breaker
    opening. Defaults to `100`.

## What it does and does not model

It runs on a **virtual clock**, so simulating half an hour of a background job
costs microseconds and no test waits for anything. Delays are nominal, with
`:jitter` switched off — the same choice `RetryOptions.window/1` and `explain/1`
make, so that all three agree and a simulation asserted in a test does not vary
between runs. Jitter changes the exact `:worst_call`, never whether a breaker
opens.

The delays come from the library's own planner, the options are resolved through
the same path `start/2` uses, and melting follows the service's `:melt` setting.
The one thing modelled rather than executed is the circuit breaker's sliding
failure window, which is what makes a virtual clock possible at all — a real
breaker counts against real time. That model is pinned against measured behavior
in the test suite, including a configuration that stays closed through twelve
consecutive failing calls.

The rate limiter and the concurrency limit are not simulated. Neither melts the
breaker, and both govern the traffic reaching a service rather than what the
service does with a failure.

Simulated calls arrive one after another, which is the worst case for opening a
breaker: concurrent callers deliver failures closer together, so a breaker that
opens here opens at least as readily under real traffic.

## Examples

    iex> %ExternalService.Simulation{opens_after: opens} =
    ...>   ExternalService.simulate(
    ...>     [circuit_breaker: [tolerate: 3], retry: [base: 100, max_attempts: 5]],
    ...>     :always_failing
    ...>   )
    iex> opens
    4

# `start`

```elixir
@spec start(service(), options()) :: :ok
```

Initializes the circuit breaker (and optional rate limiting and default retry
options) for a specific service.

The `service` is a term that uniquely identifies an external service within the
scope of an application.

## Configuration checks

These options are validated individually, and then checked *against each other* —
because the mistakes worth catching are pairs of options that are each valid and
jointly wrong. A window narrower than the failures it has to count, or ten
attempts of uncapped exponential backoff, produce a warning naming the setting to
change and a value to try.

Services declared with `use ExternalService` are checked **at compile time**, so
the warning carries a file and a line and fails a build compiled with
`--warnings-as-errors`. Services started through this function are checked here,
which is also what covers child-spec overrides, since those are runtime values.

Combinations that cannot work at all — rather than merely being unwise — raise
instead. Set how findings are reported with:

    config :external_service, on_suspicious_config: :warn   # | :raise | :ignore

`:warn` is the default. `:raise` is worth setting in a test environment, where a
suspicious configuration is better as a failure than as a log line. `:ignore` is
for a configuration you have decided is right despite what the checks make of it.

## Options

* `:circuit_breaker` (`t:keyword/0`) - Circuit breaker configuration. The default value is `[]`.

  * `:tolerate` - Number of failures tolerated within the `:within` window before the breaker opens. What counts as one failure is set by `:melt`, and defaults to one failing **call** — so `tolerate: 3` means three dead calls, whatever `:max_attempts` is. `:infinity` installs no breaker at all: it never opens, holds no state, and cannot be combined with `:fault_injection`. The default value is `10`.

  * `:melt` - What one unit of `:tolerate` counts. `:per_call` (the default) melts the breaker once per call, when its retrying gives up, so `:tolerate` is denominated in calls and is independent of `:max_attempts`. `:per_attempt` melts on every failing attempt, which is how versions before 3.0 behaved: a single call with `max_attempts: 5` then contributes up to 5, and `:tolerate` cannot be tuned independently of the retry options. `:per_call` requires retrying to be bounded — see the note on unbounded retries below. The default value is `:per_call`.

  * `:within` - Length of the failure-counting window, in milliseconds. `:auto` (the default) sizes the window against the retry options instead, since how long it takes `:tolerate` failures to arrive depends on how long a failing call takes. It is a floor, never narrower than the 10 seconds it replaces — see [Sizing the window](tuning.md) for when to set it yourself. The default value is `:auto`.

  * `:reset` (`t:pos_integer/0`) - Milliseconds to wait before the breaker resets (closes) after it has opened. The default value is `60000`.

  * `:fault_injection` (`t:float/0`) - If set to a rate between `0.0` and `1.0`, randomly fails that fraction of calls. Intended for testing how dependents behave when this service is degraded.

  * `:backend` - The circuit breaker implementation to use, either as a module or as a `{module, options}` tuple whose options are passed through to that backend. Defaults to the node-local `:fuse`-based breaker.

* `:rate_limit` (`t:keyword/0`) - Optional rate-limiting configuration. Omit for no rate limiting.

  * `:limit` - Required. Maximum number of calls allowed within each `:per` window. `:infinity` installs no limiter at all — calls pass straight through, exactly as if `:rate_limit` had been omitted. It is meant for overriding a configured limit (a child spec override cannot remove a key); to have no rate limiting in the first place, omit `:rate_limit`. `:per` is still required, and ignored.

  * `:per` (`t:pos_integer/0`) - Required. Length of the rate-limiting window, in milliseconds.

  * `:wait` - How long a throttled call may wait for the rate limit to admit it. `:infinity` waits as long as it takes, `false` never waits, and an integer is a millisecond budget for the whole call. When the budget runs out the call is not made and an `ExternalService.RateLimited` error is returned (or raised by `call!/3`). Defaults to one window — `:per`, capped at 5 seconds — which absorbs a burst without converting sustained throttling into unbounded latency. Use `:infinity` for background work and pipelines, where sleeping is how back-pressure propagates upstream.

  * `:backend` - The rate limiter implementation to use, either as a module or as a `{module, options}` tuple whose options are passed through to that backend. Defaults to `ExternalService.RateLimiter.Local`. Use `ExternalService.RateLimiter.Hammer` for a limit shared across a cluster.

* `:concurrency` (`t:keyword/0`) - Optional concurrency limit (the bulkhead pattern). Omit for no limit. See `ExternalService.Concurrency`.

  * `:limit` (`t:pos_integer/0`) - Required. Maximum number of calls allowed to be in flight against the service at once.

  * `:reclaim_after` (`t:pos_integer/0`) - Required. Milliseconds after which a held slot is considered abandoned and may be reused. A slot is normally released as soon as the call finishes, but a caller killed from outside — including by the ordinary `:shutdown` a supervisor sends while draining — never runs its release. This bounds how long such a slot is lost. Required rather than defaulted: it must exceed the longest legitimate call, which depends on the timeout configured in your HTTP client. Setting it too low silently admits more than `:limit`.

  * `:wait` - How long a call may wait for a slot before being shed. `false` (the default) sheds immediately; an integer is a millisecond budget. A short budget absorbs bursts without allowing a pile-up, since waiting callers hold no slot and are bounded by arrival rate times the budget. Unlike the rate limiter's `:wait`, `:infinity` is not accepted — an unbounded wait is the pile-up a concurrency limit exists to prevent. The default value is `false`.

* `:retry` - Default retry options for the service, used by `call/2`. See `ExternalService.RetryOptions` for the available keys. The default value is `[]`.

* `:sleep_function` (function of arity 1) - Overrides the function used whenever a call waits — between retry attempts, while rate limited, and while waiting for a concurrency slot. Called with the number of milliseconds to wait. Defaults to `Process.sleep/1`; overriding it lets tests exercise backoff and throttling without waiting for them.

# `stop`

```elixir
@spec stop(service()) :: :ok
```

Stops the fuse for a specific service.

Stopping is idempotent: it is safe to call on a service that was never started
or has already been stopped.

---

*Consult [api-reference.md](api-reference.md) for complete listing*