Retries

Many failures when calling an external service are transient: a momentary timeout, a brief overload, a connection reset. The simplest effective response is to try again — perhaps after a short backoff. ExternalService automates this using the retry library, exposing its flexibility through the ExternalService.RetryOptions struct.

Triggering a retry

Inside the function you pass to call, you signal that a retry should happen by returning either:

• the atom :retry, or
• a tuple {:retry, reason}, where reason is any term.

Anything else is a success and is returned to the caller as-is — including the function's own {:error, reason} results. You decide what is retriable:

call fn ->
  case HTTP.post(url, body) do
    {:ok, %{status: 200} = resp}            -> {:ok, resp}
    {:ok, %{status: s}} when s in 500..599  -> {:retry, s}   # retry server errors
    {:ok, %{status: 429}}                   -> :retry        # retry throttling
    {:ok, %{status: 4xx}} = resp            -> resp          # client error: don't retry
    {:error, reason}                        -> {:error, reason}
  end
end

Each retry melts the service's circuit breaker, so a sustained run of retries will eventually open it. See Circuit breakers.

Retrying on the return value with :retry_on

Returning :retry works when you control the function body. When you're calling an existing function that already returns its own result shape — and you'd rather not wrap it just to translate that shape into :retry — the :retry_on retry option takes a predicate that is run on the return value. When the predicate returns a truthy value, the call is retried exactly as if the function had returned {:retry, result}: the result becomes the retry reason, and the circuit breaker melts.

# Retry any 5xx response from an unmodified client function.
retry: [retry_on: &match?({:error, %{status: s}} when s in 500..599, &1)]

call fn -> Stripe.charge(params) end

If retries are exhausted, the matched result is carried as the reason on ExternalService.RetriesExhausted (and in the [:external_service, :call, :retry] telemetry metadata). An explicit :retry / {:retry, reason} return from the function always takes precedence over the predicate.

Configuring retries

Retry behavior is described by ExternalService.RetryOptions. You can supply it as the service's default (the :retry option to use ExternalService / start/2), or per call as a keyword list or struct:

# Service default
use ExternalService,
  retry: [max_attempts: 5, backoff: :exponential, base: 100, jitter: true]

# Per-call override (keyword list)
call [max_attempts: 2, backoff: :linear, base: 50], fn -> work() end

# Per-call override (struct)
call %ExternalService.RetryOptions{max_attempts: 2}, fn -> work() end

When you use the two-argument call/2 (no options), the service's default :retry options apply.

The options

Options are validated when the struct is built; an invalid value raises NimbleOptions.ValidationError with a helpful message.

Backoff strategies

Exponential backoff grows the delay multiplicatively, starting from :base. This is the right default for most services: it backs off quickly when a service is struggling.

retry: [backoff: :exponential, base: 100]
# delays grow ~100ms, 200ms, 400ms, 800ms, ...

Linear backoff grows the delay by :factor each time, starting from :base:

retry: [backoff: :linear, base: 100, factor: 1]
# delays grow ~100ms, 200ms, 300ms, 400ms, ...

Bounding retries

:max_attempts defaults to 5, so retrying always stops on its own. There are two bounds and they compose:

• :max_attempts — a count, defaulting to 5. max_attempts: 5 means at most five attempts total (the first try plus four retries).
• :expiry — a time budget in milliseconds for the retrying. Backoff delays are used as-is while they fit; the one that would overshoot the budget is trimmed instead, so the final attempt starts exactly at the deadline rather than past it. It is checked between attempts, so it bounds when the next attempt starts, never how long the current one runs — see Nothing here bounds a single attempt.

You can use either or both; whichever is reached first stops the retries. When the bound is hit without success, call/3 returns {:error, %ExternalService.RetriesExhausted{}} (and call!/3 raises it).

# Stop after 5 attempts OR 5 seconds, whichever comes first.
retry: [max_attempts: 5, expiry: :timer.seconds(5), backoff: :exponential, base: 100]

Nothing here bounds a single attempt

:expiry reads like a wall-clock budget for the whole call. It isn't, and the difference matters as soon as your function is slow rather than fast-failing. Both bounds are evaluated between attempts — when the retry machinery decides whether to make another one — so neither can interrupt an attempt already running.

Measured with retry: [max_attempts: 4, expiry: 100] against a function that sleeps 300ms and then returns :retry:

attempts made: 2
total elapsed: 621ms

The 100ms budget stopped the third attempt, but the call still took six times the budget, because each attempt ran to completion first. Push that further — a function that never returns — and :expiry is never evaluated at all, so the call hangs forever no matter what you set.

ExternalService imposes no timeout on your function. Bounding a single attempt is the caller's job; see When the service hangs.

Unbounded retries are a choice you have to make

Retrying without a count bound is available, but it has to be asked for:

ExternalService.start(:my_service, retry: [max_attempts: :infinity])

That is the right answer for some work — a background job that should keep trying until it succeeds, where there is nowhere else for the failure to go. It is almost never the right answer in a request path, where a call that never returns is worse than one that fails.

If you do reach for it, pair it with an :expiry, or at least a :cap so the delay between attempts doesn't grow without limit. And read the warning above first: the circuit breaker will not reliably stop the retrying for you.

Capping the delay

Exponential backoff grows without bound. :cap puts a ceiling on any single delay so you don't end up waiting minutes between attempts:

retry: [backoff: :exponential, base: 100, cap: :timer.seconds(2)]
# delays grow 100, 200, 400, 800, 1600, 2000, 2000, ... (capped at 2s)

Jitter

When many processes retry on the same schedule, they retry in lockstep and slam the recovering service all at once — the thundering herd. Jitter randomizes each delay to spread them out:

retry: [backoff: :exponential, base: 100, jitter: true]   # ±10%
retry: [backoff: :exponential, base: 100, jitter: 0.25]   # ±25%

Enabling jitter is good practice for any service with many concurrent callers.

Retrying on raised exceptions

By default, raised exceptions are not retried — they propagate straight to the caller. This is a deliberate 2.0 change (see issue #7): retrying every RuntimeError by default tended to mask real bugs.

If a particular exception genuinely indicates a transient condition worth retrying, list its module in :retry_exceptions:

retry: [retry_exceptions: [MyApp.TransientError, DBConnection.ConnectionError]]

Now a raised MyApp.TransientError triggers a retry just like a :retry return value would, and it melts the circuit breaker. Exceptions not in the list still propagate untouched and leave the breaker alone — :retry_exceptions governs both retrying and whether a raised exception counts against the breaker.

If retries run out while retrying an exception, that original exception is re-raised — with its original stacktrace, still pointing at the code that raised it — rather than being converted into an ExternalService.RetriesExhausted.

Deciding per exception rather than per type

A module list settles the question by type, which is the wrong grain when the same exception is transient in one instance and permanent in another — an HTTP client that raises one error struct for every status, say. Give :retry_exceptions a predicate instead and it is run on the exception itself:

retry: [
  retry_exceptions: fn
    %MyApp.HTTPError{status: status} -> status >= 500
    _other -> false
  end
]

A truthy return retries and melts the breaker; anything else propagates the exception untouched, exactly as an unlisted module would. The predicate replaces the list rather than supplementing it, so fold any module checks you still want into it:

retry_exceptions: fn
  %MyApp.HTTPError{status: status} -> status >= 500
  error -> is_struct(error, DBConnection.ConnectionError)
end

use ExternalService, retry: [retry_exceptions: fn error -> ... end]
# ** (ArgumentError) cannot inject attribute @__external_service_opts__ into
#    function/macro because cannot escape #Function<...>

This pairs well with Errata, whose errors classify themselves — an error type can decide from its own :reason or :context and answer through Errata.retryable?/1:

defmodule MyApp.Retry do
  require Errata

  def retryable_error?(error), do: Errata.is_error(error) and Errata.retryable?(error)
end

# ...then, in the service configuration:
retry: [retry_exceptions: &MyApp.Retry.retryable_error?/1]

See Using Errata in Your Application for both boundaries together, and for the difference between an error being retryable and a call being safe to repeat.

Putting it together

A solid default for an HTTP-style dependency:

use ExternalService,
  circuit_breaker: [tolerate: 15, within: :timer.seconds(5), reset: :timer.seconds(5)],
  retry: [
    backoff: :exponential,
    base: 100,
    cap: :timer.seconds(2),
    max_attempts: 5,
    expiry: :timer.seconds(10),
    jitter: true
  ]

This retries transient failures with jittered exponential backoff, never waits more than 2 seconds between attempts, gives up after 5 attempts or 10 seconds, and opens the circuit breaker on the third consecutive fully-failing call.

The breaker settings are sized against the retry settings rather than picked independently, and that is not optional. Five attempts spread over a ~1.5 second retry window means :within has to be wider than 1.5 seconds for the melts to accumulate at all, and :tolerate has to be about 3 × max_attempts for three calls to be what trips it. Get that wrong and the breaker never opens: see Sizing the breaker for the measurement, and the Tuning guide for choosing all of these together.