# `Perplexed.Search.SearXNG.Pool`
[🔗](https://github.com/agoodway/perplexed/blob/v0.2.0/lib/perplexed/search/searxng/pool.ex#L1)

Health, speed and rate-limit state for SearXNG instances, learned from the
searches this node makes. One pool runs under the `:perplexed` application;
tests and hosts can start more with `:name`.

## Ranking

Each instance keeps exponentially weighted moving averages (EWMA) of its
response time and failure rate, and is scored

    latency_ewma_ms * (1 + failure_penalty * failure_ewma) * jitter

lowest first. An instance not measured yet scores its prior: the
searx.space median when it was `seed/2`ed, else `:prior_ms`. `:jitter`
multiplies each score by a random factor in `1 +/- jitter`, so the
fastest few share the load instead of one taking every search.
`:min_interval_ms` ranks an instance used that recently after every
rested one, least recently used first.

## Circuit breaker

Each instance is a small state machine:

    :closed --refusal--> :open --cooldown timer--> :half_open
       ^                   ^                          |
       |                   +--------trial fails-------+
       +------------------------trial answers---------+

A refusal (HTTP 429 or 503, a redirect from SearXNG's limiter, 403/418, a
bot challenge, a timeout, a transport error) opens the circuit. The
cooldown is the server's own `Retry-After` / `RateLimit-Reset` when it
sent one, otherwise `:cooldown_ms` doubled per consecutive refusal, capped
at `:max_cooldown_ms` either way. When it ends a timer moves the instance
to `:half_open`, where it hands out a single trial lease: an answer closes
the circuit, a refusal opens it again for twice as long. An open instance
is never checked out. An empty or off-topic answer is a soft failure: the
instance is reachable, so the circuit closes, but its failure average
rises, and `:soft_strikes` (default 3) in a row open the circuit like a
refusal: an instance that keeps answering off-topic is skipped for a
while however fast it is.

## Leases

`checkout/3` picks and reserves instances in one step inside the server,
so concurrent searches spread out instead of racing onto the same
instance. The pool monitors the caller; if it dies, its leases are
released. `record/4` reports the outcome and returns the lease.

Rankings and stats are read straight from ETS, so inspection never waits
on the process. Instances idle longer than `:forget_after_ms` are swept
out.

Every state change emits `[:perplexed, :searxng, :circuit]` with metadata
`instance`, `from`, `to` and `cooldown_ms`.

# `circuit`

```elixir
@type circuit() :: :closed | :open | :half_open
```

# `lease`

```elixir
@type lease() :: {String.t(), reference()}
```

# `outcome`

```elixir
@type outcome() ::
  {:ok, non_neg_integer()}
  | {:error, atom()}
  | {:error, atom(), non_neg_integer() | nil}
```

# `stats`

```elixir
@type stats() :: %{
  circuit: circuit(),
  latency_ms: float() | nil,
  failure_rate: float(),
  successes: non_neg_integer(),
  failures: non_neg_integer(),
  consecutive_failures: non_neg_integer(),
  consecutive_soft: non_neg_integer(),
  cooldown_until: integer() | nil,
  in_flight: non_neg_integer(),
  last_outcome: atom() | nil,
  last_used_at: integer() | nil,
  prior_ms: number() | nil,
  seeded_at: integer() | nil
}
```

# `checkout`

```elixir
@spec checkout([String.t()], non_neg_integer(), keyword()) :: [lease()]
```

Reserves up to `count` of `candidates`, best first, skipping open
circuits and half-open ones whose trial is already out. Returns the
leases to pass to `record/4`.

Options: `:pool`, `:jitter` (0 to 1, default 0), `:min_interval_ms`
(default 0), `:prior_ms` (default 2500), `:failure_penalty`
(default 3.0) and `:random` (a zero-arity function
returning a float in [0, 1), for tests).

# `child_spec`

Returns a specification to start this module under a supervisor.

See `Supervisor`.

# `rank`

```elixir
@spec rank([String.t()], keyword()) :: [String.t()]
```

The instances in `candidates` that `checkout/3` would hand out, best
first, without reserving them. Takes `checkout/3`'s options.

# `record`

```elixir
@spec record(lease(), outcome(), non_neg_integer(), keyword()) :: :ok
```

Reports how a leased search went and releases the lease. `outcome` is
`{:ok, hits}`, `{:error, reason}`, or `{:error, reason, retry_after_ms}`
when the instance said how long to wait.

# `reset`

```elixir
@spec reset(keyword()) :: :ok
```

Forgets everything the pool learned and cancels its timers.

# `seed`

```elixir
@spec seed([{String.t(), number()}], keyword()) :: :ok
```

Sets prior response times (ms) for instances not measured yet, such as
searx.space medians from `Perplexed.Search.SearXNG.Directory`.

# `snapshot`

```elixir
@spec snapshot(keyword()) :: [{String.t(), stats()}]
```

Every instance the pool knows, with its stats, best first.

# `start_link`

```elixir
@spec start_link(keyword()) :: GenServer.on_start()
```

Starts a pool.

Options: `:name` (default `Perplexed.Search.SearXNG.Pool`), `:cooldown_ms`
(default 60000), `:max_cooldown_ms` (default 1800000),
`:forget_after_ms` (default 21600000) and `:sweep_every_ms`
(default 600000) and `:soft_strikes` (default 3).

# `stats`

```elixir
@spec stats(String.t(), GenServer.name()) :: stats()
```

The stats for `instance` (fresh ones when it has none yet).

---

*Consult [api-reference.md](api-reference.md) for complete listing*