How Scorch works

A small HTTP boundary separates callers from one bounded runtime that finds, fetches, renders, extracts, and follows public web content.

Process boundary

Scorch is a six-crate Rust workspace that produces two executables. The split is architectural, not cosmetic: client operations always cross HTTP, while runtime policy stays inside the service.

Client process

scorch

CLI, benchmark, and MCP adapter. It only speaks HTTP.

Service process

scorchd

Axum API, policy, metasearch, browser runtime, extraction, and crawl jobs.

Public web

Guarded egress

Pinned direct fetch or isolated browser work through the validating proxy.

Crate	Responsibility
`metasearch`	Engine adapters, concurrent routing, normalization, cache, circuit breakers, and reranking.
`scorch-types`	Shared HTTP request and response contracts.
`scorch-engine`	Network policy, validating proxy, fetch, browsers, extraction, search, mapping, and crawl jobs.
`scorch-api`	Axum routes, strict request parsing, response headers, and HTTP error mapping.
`scorch-server`	The `scorchd` executable, service configuration, logging, and lifecycle.
`scorch-cli`	The `scorch` HTTP client, benchmark command, and API-backed MCP adapter.

The client does not link scorch-engine, scorch-api, metasearch, Obscura, or Chromium. This prevents a CLI or MCP caller from accidentally bypassing service policy.

Scrape pipeline

Parse a strict contract. Unknown JSON fields are rejected and option ranges are checked before work starts.
Validate the target. Scorch accepts only HTTP and HTTPS URLs without embedded credentials, blocks unsafe ports, and resolves the host against local and special-purpose address policy.
Direct fetch first. Redirects are handled manually, each hop is revalidated, DNS answers are pinned to the outbound connection, and response bytes are capped.
Choose an engine. Static content stays on the direct path. JavaScript-dependent content, explicit rendering, and screenshots select an allowed browser.
Extract. The resulting HTML becomes requested Markdown, HTML, text, links, metadata, and optionally a PNG screenshot data URI.
Return only requested large fields. The response identifies the final URL, selected engine, elapsed time, metadata, warnings, and formats.

RENDER / AUTO

Fetch first, then render only when needed.

RENDER / ALWAYS

Force the selected allowed browser.

RENDER / NEVER

Return direct-fetch extraction only.

Browser rendering

Obscura is the default renderer. It runs in-process as a Rust library—there is no Obscura sidecar, executable, CDP server, or subprocess. Chromium remains an optional compatibility backend.

Each Obscura request receives a fresh browser context, page, and V8 state. Scorch does not pool contexts or pages across callers because that could leak cookies, authenticated cache entries, DOM state, or JavaScript state. Browser concurrency is bounded, and one absolute deadline covers validation, queueing, navigation, explicit waiting, serialization, and screenshots.

Stealth transport is enabled by default. It provides Chrome-like TLS and HTTP behavior, ordered headers, aligned user-agent/client-hint/JavaScript identity, and blocking of common tracker or fingerprinting domains. It does not hide the source IP, provide anonymity, guarantee CAPTCHA bypass, or weaken SSRF controls.

A request may name obscura or chromium, but only the service allowlist decides whether that choice is legal. Backend selection cannot escape SCORCH_ALLOWED_BROWSERS.

Metasearch

Metasearch is Scorch's only search provider. Source engines are internal adapters and cannot be selected by API, CLI, or MCP requests. The server may allow Bing, Brave, DuckDuckGo, Google, Naver, and Wikipedia; Brave and Google require operator credentials.

Start every enabled and healthy engine concurrently under per-engine limits.
Accept the first useful result, then hold a short collection window for additional agreement.
Cancel unfinished work rather than letting one slow engine determine total latency.
Normalize URLs by removing fragments and common tracking parameters.
Merge duplicates and rank with weighted reciprocal-rank fusion.
Report contributing sources and completed-engine failures in warnings.

The runtime maintains a bounded 60-second in-memory result cache and temporary circuit breakers after repeated failures. Search is best-effort: provider markup changes, rate limits, and challenges are treated as failures rather than authoritative empty results.

Mapping and crawl

Mapping is sitemap-first. Scorch checks robots-declared sitemaps, the conventional sitemap URL, bounded nested sitemap indexes, and finally links on the root page. URLs are normalized, deduplicated, path-filtered, and restricted to the requested site.

Crawling starts from mapped URLs, follows same-origin links breadth-first, applies include and exclude paths, and respects robots.txt as ScorchBot. Jobs run in bounded batches and preserve individual page failures beside successful documents.

Maximum 100 pages and discovery depth 5 per crawl.
Maximum four active crawl jobs.
Five-minute absolute crawl deadline.
32 MiB retained data per job and 128 retained jobs.
Paginated status responses, cancellation, and configurable completed-job TTL.

Security boundary

Pages, search results, browser output, sitemaps, robots files, and extracted links are untrusted data. They cannot change process configuration or trigger command execution.

Direct path

Manual redirects, DNS validation and pinning, HTTP(S)-only URLs, unsafe-port rejection, and response caps.

Browser path

The same target checks plus an embedded HTTP/CONNECT proxy that validates browser redirects and subresources.

Obscura receives only the validating proxy endpoint and retains its own private-address guard. Chromium is configured without a direct network route: HTTP, HTTPS, WebSocket, iframe, worker, redirect, and subresource connections use the proxy; QUIC and non-proxied WebRTC UDP are disabled.

Authentication is outside this boundary. The local-first API has no built-in authentication or TLS. Use loopback or an authenticated TLS gateway when serving other machines.

Runtime state

scorchd owns one Axum server, one globally bounded direct-fetch runtime, bounded isolated Obscura work, an optional lazy Chromium backend, one validating proxy, and a bounded in-memory crawl registry.

There is no durable state. A restart clears crawl jobs, metasearch cache entries, and circuit-breaker state. This keeps deployment self-contained but means clients must persist any results they need to retain.