Getting started

What you will run

A release build creates two executables with a strict process boundary:

You do not need PostgreSQL, Redis, a message broker, a separate browser service, or worker processes. Crawl state is held in bounded memory and disappears when scorchd restarts.

Install

Option A: Nix flake

Run either released executable without cloning the repository:

nix run github:Fractal-Tess/scorch/v0.7.0#scorch -- --help
nix run github:Fractal-Tess/scorch/v0.7.0#scorchd -- --help

For a declarative NixOS service and client installation, continue to the Nix installation guide.

Option B: Nix development shell

The repository flake provides Rust, CMake, Clang, libclang, pkg-config, Git, Bun, and Jujutsu. Its .envrc loads that shell automatically when using direnv:

git clone https://github.com/Fractal-Tess/scorch.git
cd scorch
direnv allow
cargo build --release --workspace

Without direnv, run nix develop before building instead.

Option C: native toolchain

Install stable Rust plus the native tools needed by embedded Obscura:

• Rust stable with Cargo, Clippy, and rustfmt
• CMake, Clang, libclang, and pkg-config
• Git

If libclang is not discovered automatically, point LIBCLANG_PATH at the directory containing the shared library before building.

export LIBCLANG_PATH=/path/to/libclang/lib
cargo build --release --workspace

The optimized binaries are written to:

target/release/scorchd
target/release/scorch

Install them somewhere on your PATH if desired:

install -Dm755 target/release/scorchd ~/.local/bin/scorchd
install -Dm755 target/release/scorch ~/.local/bin/scorch

Start the API

scorchd

At startup the service prints operational logs to stderr. Verify liveness and readiness from another terminal:

curl -sS http://127.0.0.1:33000/health
curl -sS http://127.0.0.1:33000/ready | jq

/ready reports Obscura availability and stealth mode, global concurrency, search provider, and enabled search engines.

Connect a client

The scorch client defaults to http://127.0.0.1:33000. Set one environment variable to use a different service:

export SCORCH_API_URL=https://scorch.internal.example
scorch search "rust async runtime" --limit 5

Every command also accepts --api-url:

scorch --api-url http://127.0.0.1:33000 search "web extraction"

SCORCH_API_URL configures only the client and MCP adapter. SCORCH_BIND controls where the service listens. They are intentionally separate settings.

Make a request

With the CLI

scorch scrape https://example.com --format markdown,links
scorch search "rust async runtime" --limit 5
scorch map https://example.com --limit 100
scorch crawl https://example.com --limit 20 --max-depth 2 --wait

With HTTP

curl -sS http://127.0.0.1:33000/v1/scrape \
  -H 'content-type: application/json' \
  -d '{
    "url": "https://example.com",
    "options": {
      "formats": ["markdown", "links"]
    }
  }' | jq

Every scrape executes JavaScript through embedded Obscura with its stealth transport enabled. Rendering and transport policy are fixed by the service.

Production notes

• Put authentication and TLS at the edge. Scorch does not provide either itself.
• Do not treat crawl jobs as durable data. Poll and export results before their TTL or a service restart.
• Watch stderr. Protocol and result output stays on stdout; structured operational logs stay on stderr.
• Build with release mode. Production profiles use thin LTO, one codegen unit, stripped symbols, and overflow checks.

SCORCH_LOG_FORMAT=json \
RUST_LOG=scorch=info \
SCORCH_BIND=127.0.0.1:33000 \
scorchd