Unified signal, less noise
Query allowed engines concurrently, normalize and merge URLs, then rerank agreement without exposing source-engine policy to callers.
Search · reciprocal-rank fusion
→
Search and extract the public web at Rust speed
Scorch gives applications one guarded service for search, scraping, mapping, and crawling—without a database, broker, browser farm, or worker fleet.
API · 127.0.0.1:33000 / Renderer · Obscura
scorch · local runtime
Request flowClient → HTTP API → bounded engine
obscura · stealth Binaries2
Required services0
Default browserObscura
Crawl stateEphemeral
terminal connected
$ scorch search "rust web extraction" --limit 3
provider: metasearch · elapsed: 842ms
01 Rust web scraping patterns
https://example.org/rust-scraping
02 Reliable HTML extraction
https://example.net/extraction
▊
Pipeline
01Validate target
02Fetch or render
03Extract + return
Every network path shares URL, DNS, redirect, port, and
response-size policy.
2 purpose-built binaries
0 required runtime services
6 focused Rust crates
MIT open-source license
One guarded runtime
Total visibility from source to document
One deliberate surface
A single system that finds, reads, and follows the web
Runtime and routing policy remain on the server. API, CLI, and MCP callers get a compact contract without gaining access to forbidden engines or backends.
Clean content on demand
Fetch inexpensive static pages directly. Escalate JavaScript pages and screenshots to embedded Obscura or an allowed Chromium backend.
Extract · Markdown · screenshots
→
Bounded discovery
Discover sitemaps and same-site links, then run robots-aware crawl jobs with pagination, cancellation, retained-byte limits, and expiry.
Map · breadth-first crawl
→
Why self-contained
A complete web pipeline gives applications an edge, but stitching services together creates tradeoffs
- The deployment gap
- Every queue, browser sidecar, cache, and worker adds another service to secure, observe, and recover.
- The policy gap
- Separate fetch and browser paths drift unless they share URL validation, DNS controls, redirects, limits, and deadlines.
Operationally compact
Clear boundaries from the first request
The client only speaks HTTP. The service owns execution, isolation, policy, and ephemeral crawl state.
2
CLIENT / SERVICE Strictly separated executables
1
DIRECT / BROWSER Shared guarded egress policy
0
SELF / CONTAINED Durable services required
From one release build to a local API, Scorch keeps the path short without hiding the security boundary.
Enter the reproducible environment, build two optimized binaries,
start scorchd, and connect through the CLI, JSON API, or
MCP.
Frequently asked questions
The constraints that matter before putting Scorch to work.
Does Scorch need a database or queue?+
No. Crawl jobs live in bounded memory and expire. Restarting scorchd removes them, so Scorch is not a durable crawl archive.
Can callers choose any engine or browser?+
No. Search engines are server policy. A scrape may request a backend
only when it appears in SCORCH_ALLOWED_BROWSERS.
Is authentication built in?+
Not in the local-first release. Keep the loopback bind, or place an authenticated TLS gateway in front of the API.
Does Obscura run as a sidecar?+
No. It is embedded in scorchd as a Rust library, with fresh
context, page, and JavaScript state per request.
Bring the public web closer to the systems that need it
Build Scorch, start scorchd, and make the first request
through a runtime you control.