Perplexed.Search.SearXNG.Directory (Perplexed v0.2.0)

Copy Markdown View Source

Picks public SearXNG instances from searx.space's daily health check (https://searx.space/data/instances.json).

Keeps instances on the open web (not Tor or I2P) that answered HTTP 200 and passed at least :min_success percent of searx.space's test searches, optionally only those hosted in :countries (ISO codes such as "US" or "DE", from the network searx.space resolved each instance's IPs to). Each entry carries searx.space's median search time, which seed/1 hands to Perplexed.Search.SearXNG.Pool as a prior until this node has measured the instance itself. searx.space measures from Germany, so the median is a starting point, not your latency.

probe/2 checks from here which instances are up right now, with one light request to each start page (never a search), and discover/1 does the whole job: fetch, filter, probe, keep the live ones and seed the pool. Perplexed.Search.SearXNG.Instances runs discover/1 on a timer, and mix perplexed.searxng.instances runs it from the shell. A live start page does not promise the instance will search for you, since SearXNG's limiter acts on /search; the pool's circuit breaker covers that.

Summary

Types

entry()
probe()

Functions

alive?(arg1)

Whether entry is worth using: probed alive, or not probed at all.

discover(opts \\ [])

Fetches the directory, probes what it lists, seeds the pool with the live instances and returns them, fastest first.

entries(body, opts \\ [])

Filters a decoded instances.json body; see fetch/1 for the options.

fetch(opts \\ [])

Fetches and filters the directory.

probe(entries, opts \\ [])

Checks each entry's start page concurrently and adds its :probe: :alive (a SearXNG page), :challenge (a bot check in front of it), :blocked (refused or redirected) or :down, with the round trip in ms. Live entries come back fastest first by searx.space median plus that round trip; the rest follow.

seed(entries, opts \\ [])

Seeds the pool with each entry's prior: searx.space's median search time, plus the probe's round trip from here when the entry was probed.

urls(entries)

The entries' URLs, for the :instances option.

Types

entry()

@type entry() :: %{
  :url => String.t(),
  :countries => [String.t()],
  :median_ms => non_neg_integer() | nil,
  :engines => [String.t()],
  optional(:probe) => probe()
}

probe()

@type probe() :: %{
  status: :alive | :challenge | :blocked | :down,
  latency_ms: non_neg_integer()
}

Functions

alive?(arg1)

@spec alive?(entry()) :: boolean()

Whether entry is worth using: probed alive, or not probed at all.

discover(opts \\ [])

@spec discover(keyword()) :: {:ok, [entry()]} | {:error, term()}

Fetches the directory, probes what it lists, seeds the pool with the live instances and returns them, fastest first.

Takes fetch/1's and probe/2's options, plus probe: false to skip probing (every listed instance is kept) and :pool.

entries(body, opts \\ [])

@spec entries(map(), keyword()) :: [entry()]

Filters a decoded instances.json body; see fetch/1 for the options.

fetch(opts \\ [])

@spec fetch(keyword()) :: {:ok, [entry()]} | {:error, term()}

Fetches and filters the directory.

Options

  • :countries - ISO country codes to keep, default all.
  • :min_success - lowest test-search success percentage, default 90.
  • :max_median_ms - slowest median search time to keep, default none.
  • :req_options - merged into the Req call (tests pass plug:).

probe(entries, opts \\ [])

@spec probe([entry()], keyword()) :: [entry()]

Checks each entry's start page concurrently and adds its :probe: :alive (a SearXNG page), :challenge (a bot check in front of it), :blocked (refused or redirected) or :down, with the round trip in ms. Live entries come back fastest first by searx.space median plus that round trip; the rest follow.

Options: :timeout_ms (default 5000), :max_concurrency (default 8), and the request options SearXNG searches take (:proxy, :impersonate, :req_options).

seed(entries, opts \\ [])

@spec seed([entry()], keyword()) :: :ok

Seeds the pool with each entry's prior: searx.space's median search time, plus the probe's round trip from here when the entry was probed.

urls(entries)

@spec urls([entry()]) :: [String.t()]

The entries' URLs, for the :instances option.