HTTP API

Connection

The default base URL is http://127.0.0.1:33000. All request bodies use JSON with camel-case property names and content-type: application/json.

curl -sS http://127.0.0.1:33000/health

Request objects reject unknown fields. Every response includes an x-request-id header suitable for correlation with service logs.

Health and readiness

GET /health

Process liveness. A healthy process returns HTTP 200:

{ "status": "ok" }

GET /ready

Reports runtime policy and browser availability. It returns HTTP 200 when the configured browser is available, or 503 when unavailable.

{
  "status": "ready",
  "browserAvailable": true,
  "browser": "obscura",
  "allowedBrowsers": ["obscura"],
  "obscuraStealth": true,
  "maxConcurrency": 4,
  "searchProvider": "metasearch",
  "searchEngines": ["bing", "duckduckgo", "naver", "wikipedia"]
}

Scrape

POST /v1/scrape

{
  "url": "https://example.com",
  "options": {
    "formats": ["markdown", "html", "text", "links", "metadata"],
    "render": "auto",
    "browser": "obscura",
    "timeoutMs": 30000,
    "waitForMs": 0,
    "onlyMainContent": true,
    "blockMedia": true,
    "fullPageScreenshot": false
  }
}

Screenshot output is a PNG data URI. The response engine is fetch, obscura, or chromium.

{
  "url": "https://example.com",
  "finalUrl": "https://example.com/",
  "engine": "fetch",
  "elapsedMs": 143,
  "metadata": {
    "title": "Example Domain",
    "statusCode": 200,
    "contentType": "text/html"
  },
  "markdown": "# Example Domain\n\n...",
  "links": [],
  "warnings": []
}

Search

POST /v1/search

{
  "query": "rust async runtime",
  "limit": 5,
  "country": "us",
  "language": "en",
  "scrapeOptions": null
}

Limits range from 1 to 20 and queries are capped at 512 bytes. Metasearch is the only public provider; requests cannot name source engines. Results identify their contributing sources.

{
  "query": "rust async runtime",
  "provider": "metasearch",
  "results": [
    {
      "position": 1,
      "title": "Result title",
      "url": "https://example.com/result",
      "description": "Result description",
      "sources": ["bing", "duckduckgo"]
    }
  ],
  "elapsedMs": 911,
  "warnings": []
}

Set scrapeOptions to a normal scrape-options object to enrich each result. Per-result enrichment errors are retained in that result's error field without discarding successful results.

Map

POST /v1/map

{
  "url": "https://example.com",
  "limit": 100,
  "includeSubdomains": false,
  "includePaths": [],
  "excludePaths": []
}

Limits range from 1 to 1,000. Scorch checks robots-declared and conventional sitemaps, bounded nested sitemap indexes, and root-page links. A successful map may contain no links.

{
  "url": "https://example.com",
  "links": ["https://example.com/", "https://example.com/docs"],
  "elapsedMs": 287,
  "sources": ["sitemap"]
}

Crawl

POST /v1/crawls

Starts an in-memory crawl and returns HTTP 202 with a job summary.

{
  "url": "https://example.com",
  "limit": 20,
  "maxDepth": 2,
  "concurrency": 4,
  "includePaths": [],
  "excludePaths": [],
  "scrapeOptions": {
    "formats": ["markdown"],
    "render": "auto"
  }
}

Configured defaults and ceilings:

• Page limit 20 by default, maximum 100.
• Depth 2 by default, maximum 5.
• Concurrency 4 by default and no higher than global concurrency.
• Five-minute absolute job deadline and 32 MiB retained data per job.
• 128 retained jobs, four active crawls, and 15-minute completed-job TTL by default.

GET /v1/crawls/{id}

GET /v1/crawls/550e8400-e29b-41d4-a716-446655440000?cursor=0&pageSize=10

Page sizes range from 1 to 50. Status values are queued, running, completed, cancelled, and failed. Follow nextCursor until it is absent.

DELETE /v1/crawls/{id}

Cancels active work and removes the job. Success returns:

{ "id": "550e8400-e29b-41d4-a716-446655440000", "deleted": true }

Errors

{
  "code": "invalid_request",
  "message": "invalid request: ..."
}

Security behavior

All target-bearing endpoints accept only HTTP and HTTPS URLs without embedded credentials. Scorch rejects unsafe ports and hostnames that resolve to local, private, link-local, reserved, or other special-purpose addresses.

Direct redirects are manual, limited to five, and independently revalidated. Browser connections route through an embedded validating HTTP/CONNECT proxy. Response bytes, request bodies, queueing, rendering, crawl growth, retained job data, and job lifetimes are bounded.