# `Perplexed.Scrape`
[🔗](https://github.com/agoodway/perplexed/blob/v0.2.0/lib/perplexed/scrape.ex#L1)

Scrapes a URL to Markdown by climbing a ladder of `Perplexed.Scraper`s,
cheapest first.

The default ladder is `[:direct, :impersonated, :deepseek, :firecrawl]`:
plain HTTP, then a browser TLS fingerprint, both free, then the hosted
scrapers cheapest first, DeepSeek Fetch and then Firecrawl, which renders
JavaScript and gets past most bot challenges. A hosted rung is left out
when its key is not set, so with no keys the default is the two free
rungs. Swap any rung for another scraper, for example
`[:direct, :impersonated, :browser_run]`, or for your own module
implementing `Perplexed.Scraper`.

The URL is checked by `Perplexed.UrlGuard` once, before any rung, so a
private address never reaches a hosted scraper either. Each rung's text is
judged by `Perplexed.Verdict`. After a failure the climb skips rungs whose
`c:Perplexed.Scraper.retries?/1` says they cannot help with that outcome;
impersonation, for one, does nothing for a JavaScript shell. A missing
page, a redirect into a private network, a scanned PDF needing OCR, an
encrypted PDF or an oversized body ends the climb. A file type no earlier
rung could convert (`:unsupported`, such as a PDF with no
`Perplexed.PdfConverter` configured) goes on only to rungs that read such
files, which for Firecrawl means `firecrawl_pdf: true`.

A scrape has one deadline (`:timeout`). Each rung gets only the time left,
and once it has passed the next rung is recorded as `:deadline` and the
climb stops. Every rung's cost counts, including a paid rung whose page
then failed the readability check.

# `attempts`

```elixir
@type attempts() :: [{atom(), atom()}]
```

# `result`

```elixir
@type result() ::
  {:ok, Perplexed.Document.t()}
  | {:error, {:unreadable, %{attempts: attempts(), cost: float()}}}
  | {:error, Perplexed.error()}
```

# `batch`

```elixir
@spec batch([String.t()], keyword()) :: [{String.t(), result()}]
```

Scrapes `urls` concurrently and returns `{url, result}` pairs in input order.

Each scrape runs in an unlinked task under `Perplexed.TaskSupervisor`, with
bounded concurrency and a per-URL timeout that kills the task, so a crash
or a hang becomes that URL's `{:error, {:exit, %{reason: reason}}}` instead
of taking the caller down.

## Options

  * `:max_concurrency` - default 6.
  * `:timeout` - each URL's scrape deadline (see `run/2`); the task is
    killed 1000 ms after it.
  * `:task_supervisor` - default `Perplexed.TaskSupervisor`.

Other options go to `run/2`.

# `cost`

```elixir
@spec cost(result()) :: float()
```

USD cost of a scrape result, whether it succeeded or every rung failed.

# `default_ladder`

```elixir
@spec default_ladder(keyword()) :: [Perplexed.Scraper.spec()]
```

The ladder `run/2` uses when none is given: `config :perplexed, ladder:`,
else `[:direct, :impersonated, :deepseek, :firecrawl]`, cheapest first, without the paid
rungs whose keys are not set in the settings or in `opts[:credentials]`
(so with no keys it is the two free ones).

# `run`

```elixir
@spec run(String.t(), keyword()) :: result()
```

Scrapes `url`, returning the first rung's readable `Perplexed.Document`.

When every rung fails, answers
`{:error, {:unreadable, %{attempts: attempts, cost: cost}}}`, where
`attempts` lists each rung tried with its outcome and `cost` is what those
rungs cost. A blank URL, one the URL guard refuses, or an unknown ladder
entry is `{:error, {:invalid_request, %{reason: reason, ...}}}`.

## Options

  * `:ladder` - the `t:Perplexed.Scraper.spec/0` entries to try, in order,
    default `default_ladder/0`.
  * `:timeout` - the whole scrape's deadline in milliseconds, default
    60000.
  * `:url_validator` - a `{Module, :fun, args}` URL check for this call,
    overriding the configured one (see `Perplexed.UrlGuard`).

Other options go to every scraper; per-scraper options ride on a
`{Module, opts}` ladder entry.

---

*Consult [api-reference.md](api-reference.md) for complete listing*