Fetching in depth

POST /v1/fetch turns a URL into text, metadata and provenance. This guide explains what happens in between and how to steer it.

The order of operations#

1. Normalise the URL (lower-case host, default port and fragment dropped, tracking parameters such as utm_*, gclid, fbclid removed, remaining parameters sorted). This is the cache key.
2. Cache. A fresh entry that satisfies your request answers at once, with cached: true and cost_usd: 0. The cache is shared by all projects: HTML pages stay fresh for 7 days, PDFs for 30. A stale entry is revalidated with a conditional request: if the origin answers 304 Not Modified, you get the cached copy with cached: true and revalidated: true, at no cost.
3. Safety. Private, loopback, link-local and metadata addresses are refused (blocked_destination), also after redirects. A host whose name does not exist (NXDOMAIN twice within 6 h) answers error: "domain_dead" at once for 6 h — or its archived copy, see below.
4. robots.txt is honoured for CormorantBot and * (robots_disallowed).
5. Domain memory. A domain where every tier failed in 3 consecutive fetches is skipped for a while (10 min, then 30 min, 2 h, 6 h) and answers error: "domain_unavailable" at once, with retry_after_s. Any success ends the skip. Cormorant also remembers which tier (and Web Render difficulty) worked for a domain and starts there next time.
6. Tiers run until one produces a good page.

Tiers and escalation#

The ladder starts at direct (or at a tier learned for the domain, see architecture). It escalates on network errors, HTTP 403/429/451/5xx and bot-challenge pages (Cloudflare, Akamai, DataDome, PerimeterX, Imperva, Kasada, generic interstitials). Thin main text escalates direct → browser only. archive and the paid tiers are used only when our own tiers were blocked — never for a page that is simply short. Definitive answers (404, 410, 401, …) are returned without escalation. No CAPTCHA is solved unless the operator enabled it for the project (Massive option, see operations).

Dead domains and missing pages. A host whose name does not exist (NXDOMAIN — not a temporary DNS failure such as SERVFAIL or a timeout, which stay blocked_destination with reason dns_error) twice within 6 hours is remembered as dead for 6 hours: fetches answer error: "domain_dead" at once, with a single {"tier": "dns", "reason": "domain_dead"} attempt and no tier work. When a fetch ends in domain_dead or a 404/410 and allow_archive is true (and tiers, if given, includes archive), Cormorant looks for a Wayback snapshot (≤ 365 days) and, if there is one, returns it instead: tier: "archive", archived_at, and served_from_archive_because: "domain_dead" or "not_found"; attempts still shows the 404 or the dead host. These stand-in copies are not cached. The archive's paywall rules apply (no copies for known paywalled domains or of articles marked not free, archive_blocked_paywall). Without a snapshot the answer stays domain_dead or http_404/http_410 (with an archive_not_found attempt), and only then are alternative sources looked for. Other definitive statuses (401, …) never use the archive.

render=never uses direct → archive → proxy; render=always uses browser → proxy_render.

attempts[].reason values include ok, thin_content, not_modified, challenge_<vendor>, http_<status>, timeout, tls_error, dns_error, dns_nxdomain, domain_dead, network_*, not_configured, budget_exhausted, archive_not_found, massive_captcha_rejected, massive_no_credits, paywall (teaser of a hard paywall, returned as is), paywall_skip_paid, archive_blocked_paywall, and for the alternative-source steps (tier: source) alternate_<source> (with the copy's URL in detail), syndication_candidates_<n>, open_access_none, open_access_miss, guardian_api_miss, publisher_feed_miss, publisher_feed_teaser, common_crawl_none, common_crawl_not_free, common_crawl_paywalled_domain, common_crawl_index_error, common_crawl_record_error, source_timeout. Web Render attempts carry detail with the difficulty used (low, medium, high); other attempts omit detail. Busy Web Render answers (massive_render_503) are retried once and appear as two attempts.

What you get back#

Non-HTML content#

content_kind tells you how to read text:

• json — text is the decoded document (still valid JSON);
• xml — feeds, sitemaps, XBRL…: one text node per line, title from the first <title>; external entities are never resolved;
• text — plain text, CSV, Markdown, decoded;
• binary — spreadsheets, archives, images…: text is empty, download the bytes from body_url.

All of these are successful fetches (error: null).

PDFs#

PDFs are accepted by default (accept_pdf: false turns them into error: "pdf_not_accepted"). Text is extracted per page; use pdf_text_pages to cite page numbers. Large reports may need a higher max_bytes (default 15 MB, at most 50 MB); a bigger body is error: "too_large".

Exact bytes and provenance#

Every fetch stores the exact bytes it retrieved, content-addressed by sha256, and returns body_url (null when object storage is unavailable). body_kind says what the hash covers:

• origin: the HTTP entity body the origin (or the archive, for tier = archive) sent, after content-encoding (gzip/br) is removed — the bytes a plain HTTP client would hand you;
• rendered_dom: the document serialized by a browser (browser, proxy_render) after scripts ran — use it for text, not as proof of the origin's bytes. Non-HTML documents a browser downloads (PDFs) are origin.

GET /v1/bodies/{sha256} (same Bearer key) streams the bytes with the origin Content-Type, Content-Length, ETag: "<sha256>", X-Body-Kind and Cache-Control: immutable; If-None-Match gives 304; HEAD returns the headers only. A project can download only bodies it has retrieved itself (fresh or from the shared cache); anything else is 404. Bodies are kept while a cache entry points to them or a project retrieved them within the last 30 days. Downloading a body (Python and TypeScript continue the Quickstart client):

curl -sS "$BODY_URL" -H "Authorization: Bearer $CORMORANT_API_KEY" -o report.pdf

page = client.post(
    "/v1/fetch", json={"url": "https://www.example.com/annual-report-2025.pdf"}
).json()
if page["body_url"]:
    with client.stream("GET", page["body_url"]) as response:
        response.raise_for_status()
        with open("report.pdf", "wb") as out:
            for chunk in response.iter_bytes():
                out.write(chunk)

const bytes = await fetch(page.body_url, { headers }).then((r) => r.arrayBuffer());

For citations, keep final_url, retrieved_at, sha256 and body_kind with the quote. For tier = archive, final_url is the Wayback Machine copy and archived_at the snapshot time.

Cache semantics#

• cache_ttl_s is the maximum age you accept. 0 bypasses the cache and forces a fresh fetch.
• A cache hit is used only if it satisfies your request: render: "always" needs a rendered copy and render: "never" an origin copy; HTML text must reach your min_text_chars; a cached body larger than your max_bytes answers too_large, exactly as a fresh fetch would; with tiers set, hits from other tiers are not used.
• Best-effort pages (every tier gave thin text) are never cached.
• Hits cost nothing, even if the page originally came from a paid tier.

Per-request controls#

A cost-capped, latency-capped fetch for an agent:

{
  "url": "https://www.holcim.com/investors",
  "allow_paid": false,
  "timeout_s": 45,
  "include_html": false
}

A filing from a service that requires a contact User-Agent:

{
  "url": "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000320193",
  "user_agent": "Example Research contact@example.org",
  "tiers": ["direct"],
  "render": "never"
}

Errors#

The fetch-level error codes, what they mean and whether to retry them are in Errors & retries.