# `Perplexed.Scraper`
[🔗](https://github.com/agoodway/perplexed/blob/v0.2.0/lib/perplexed/scraper.ex#L1)

The behaviour of one scrape tier, and the registry of built-in tiers.

A scraper turns one URL into a `Perplexed.Document` and, when it read HTML,
the raw markup (so `Perplexed.Verdict` can spot challenge pages and
JavaScript shells). Failures are plain outcome atoms such as `:blocked`,
`:not_found` or `:timeout`.

Built-in scrapers, referenced by name in a `Perplexed.Scrape` ladder:

| Name            | Module                           |
| --------------- | -------------------------------- |
| `:direct`       | `Perplexed.Scraper.Direct`       |
| `:impersonated` | `Perplexed.Scraper.Impersonated` |
| `:browser_run`  | `Perplexed.Scraper.BrowserRun`   |
| `:firecrawl`    | `Perplexed.Scraper.Firecrawl`    |
| `:deepseek`     | `Perplexed.Scraper.DeepSeek`     |

Any module implementing this behaviour can stand in a ladder too, as
`Module` or `{Module, opts}`.

# `result`

```elixir
@type result() ::
  {:ok, Perplexed.Document.t(), raw_html :: String.t() | nil} | {:error, atom()}
```

# `rung`

```elixir
@type rung() :: {name :: atom(), module(), keyword()}
```

# `spec`

```elixir
@type spec() :: atom() | module() | {module(), keyword()}
```

# `configured?`
*optional* 

```elixir
@callback configured?(keyword()) :: boolean()
```

Whether the scraper can run with `opts` and the current settings, such as
a paid one whose API key is set. Optional; defaults to true. The default
ladder leaves out built-in rungs that are not configured.

# `cost`
*optional* 

```elixir
@callback cost(Perplexed.Document.t(), keyword()) :: float()
```

USD cost of producing `document` with the rung's `opts`. Optional;
defaults to 0 (free tiers).

# `retries?`
*optional* 

```elixir
@callback retries?(outcome :: atom()) :: boolean()
```

Whether this scraper is worth trying after an earlier tier failed with
`outcome`. Optional; scrapers that leave it out are always tried.

# `scrape`

```elixir
@callback scrape(String.t(), keyword()) :: result()
```

# `configured?`

```elixir
@spec configured?(rung(), keyword()) :: boolean()
```

Whether the rung's scraper can run with the call's `opts` (see
`c:configured?/1`), such as per-call `credentials:`.

# `cost`

```elixir
@spec cost(rung(), Perplexed.Document.t()) :: float()
```

USD cost of `document` produced by the rung.

# `resolve`

```elixir
@spec resolve(spec()) :: {:ok, rung()} | {:error, {:invalid_request, map()}}
```

Resolves a ladder entry to `{:ok, {name, module, opts}}`, or an
`:invalid_request` error for anything that is not a built-in name or a
module implementing `c:scrape/2`.

# `retries?`

```elixir
@spec retries?(rung(), atom()) :: boolean()
```

Whether the rung's scraper should be tried after an earlier tier failed
with `outcome`. A scraper without `c:retries?/1` is tried after anything
except `:unsupported` (a file type such as a PDF no earlier tier could
convert), which only a scraper that says it reads such files is tried
after.

# `time_left`

```elixir
@spec time_left(keyword(), pos_integer()) :: pos_integer()
```

Milliseconds a scraper may still spend: `default`, capped by the time left
before the scrape's `:deadline` (a monotonic millisecond time that
`Perplexed.Scrape.run/2` puts in the options). At least 1.

---

*Consult [api-reference.md](api-reference.md) for complete listing*