Changelog

Changes to the Cormorant API, newest first. Within v1, fields are only ever added — nothing is renamed or removed. The API contract is the reference for every field mentioned here.

Unreleased#

SEC EDGAR usage#

SEC EDGAR requests (/v1/sec/... and fetches of SEC hosts) are metered as kind sec in new billing categories sec and sec_cached (billing_category, GET /v1/usage?category=, GET /v1/requests?kind=sec). GET /v1/usage adds a sec block, keys[] adds sec_requests. SEC has no provider cost and the standard plans do not price it. For key limits, SEC requests count as fetch.

SEC EDGAR endpoints#

A new guide, SEC EDGAR via Cormorant, and MCP tools cormorant_sec_company, cormorant_sec_filings and cormorant_sec_document.

New /v1/sec/... endpoints (see the API contract): GET /v1/sec/companies/resolve (ticker, CIK or name → company), GET /v1/sec/companies/{cik}/submissions (normalised filings, including the older filing pages), GET /v1/sec/companies/{cik}/filings/latest (latest 10-K/20-F/40-F), GET /v1/sec/filings/{cik}/{accession} (documents of a filing), POST /v1/sec/documents/fetch (a filing document, same answer as POST /v1/fetch), GET /v1/sec/companies/{cik}/facts (XBRL company facts) and GET /v1/sec/search (EDGAR full-text search). New errors not_found and invalid_identifier.

SEC EDGAR policy for fetches#

Fetches of www.sec.gov, data.sec.gov and efts.sec.gov follow SEC's fair-access rules automatically: Cormorant sends its own declared identity as User-Agent (a client user_agent/accept_language is ignored for these hosts — do not send your own), only the direct tier is used (never the browser, archive, paid tiers or alternative sources), and one cluster-wide rate limit (default 8 requests/s, at most SEC's 10) is shared by all clients. New errors rate_limited and sec_unavailable (both with retry_after_s, retried automatically in batches), sec_not_configured and redirect_not_allowed. Filing documents are cached for a year, company_tickers and submissions for 12 h (revalidated with conditional requests), full-text search for 1 h. /v1/status lists a sec component (success_rate, throttle_wait_p95_ms, backoff_active, rate_limit_rps).

Company data#

New product area: the company register (company/v1). See Company data.

• GET /v1/companies/search, GET /v1/companies/{id} (by cmp_… or scheme:value, with max_age, refresh, include, sources), GET /v1/companies/{id}/history and the sub-resources events, people, ownership, relationships, filings, financials, sources.
• POST /v1/companies/resolve, batch jobs of kind company (POST /v1/companies/batch, /batch/csv, results also as GET /v1/companies/batch/{job_id}.csv).
• GET /v1/companies/changes and signed company.changed webhooks (POST/GET/DELETE /v1/companies/watches).
• GET /v1/companies/sources, GET /v1/schemas/company (JSON Schema).
• First source: GLEIF (LEI records, register-id crosswalk, parents).
• Metered as the new billing categories and meters company and company_cached (answered from the register without a source refresh); key limits and quotas accept the category company; GET /v1/usage adds a company block when there is company usage and company_requests per key.
• Status: the new group company_data lists one component per active source (company_<source>); components carry group.
• MCP: cormorant_company_search, cormorant_company_get, cormorant_company_resolve.

North Data#

• Transport policy for northdata.com / northdata.de (also for POST /v1/fetch of these hosts): fetched only through Massive — the residential proxy first (Chrome-consistent headers; a sticky session when a session_key is given, retried once with a fresh session after a proxy error), then Web Render when the proxy was blocked. Never from Cormorant's own egress (no direct, browser, archive or alternative sources); robots.txt is read through the proxy too. allow_paid: false (or a tiers list without a Massive tier) answers error: paid_tier_required, an exhausted budget error: budget_exhausted. A cluster-wide throttle for the whole site (default 2 in flight, 1 s between starts). Profile pages are served from the cache for 24 h, then revalidated (entries kept 7 days).
• Challenge detection also recognises AWS WAF, Sucuri and "unusual traffic" pages. A Massive 407 (credentials refused) pauses the residential tier cluster-wide for 5 minutes; 452 is reported as massive_content_policy.
• Profile parser v3 (parser_version: "3"): parses HTML and the Markdown Massive Web Render sometimes returns (entity lists included); legal_form from the company's own name or legal-form block; sanitised JSON-LD names; section entities only from the section's own list/table (never footer or navigation links); entity links as on the page (url) and absolute (absolute_url); new optional fields path_country, lei, industry, and officer role/since/until when the page shows them. Network graph parsing and merging (graph companies → related, persons → officers).
• New endpoints POST /v1/northdata/suggest, /discover, /profile and /graph (authenticated, metered as fetches). Discovery: suggest first (name similarity ≥ min_similarity, default 70), else Cormorant's own web search (four query templates, URL score ≥ 0.35, early stop at 0.75). Profiles come with fetch provenance (body_url, sha256, body_kind, tier, attempts, cost_usd, parser_version) and, with include_graph (default), the merged network graph. Suggest and graph answers are cached 24 h. Errors are HTTP 200 with error: not_found, blocked, rate_limited (+ retry_after_s), all_tiers_failed, parse_failed, invalid_identifier, paid_tier_required, budget_exhausted, robots_disallowed, timeout.

Account admin keys and additional keys#

API keys now have a role, fixed at creation and visible in the prefix: account admin keys start with cma_, additional (standard) keys with cmr_ as before. Every authenticated /v1 response carries X-Cormorant-Key-Role: account_admin|standard. Existing keys are standard keys.

Usage per API key#

Requests are attributed to the API key that made them (batch items to the key that created the job). GET /v1/usage?group_by=key adds keys (usage per key of the project), and GET /v1/account adds keys (the month's usage and share of the charges per key of the account). Keys appear by prefix and label only.

Privacy fix#

• Removed massive_reported_account_wide from GET /v1/usage: it was the paid provider's usage across all customers and is not customer data. This is the one exception to the add-only rule: it should never have been exposed.

Cache revalidation#

• New fetch field revalidated: true when an expired cache entry was confirmed unchanged by the origin (conditional GET answered 304; attempt reason not_modified). The cached copy is returned with cached: true and no cost.
• Only cache entries that satisfy the request are revalidated, and new validators from a 304 are stored.

Usage, charges and keys from the aggregates#

• GET /v1/usage is rebuilt on the usage aggregates (refreshed every 5 minutes; as_of, final, and final per day): group_by takes key, project, category and day, combinable; filters key, project, category; new totals and groups with failures by error, latency p50/p95, bytes, paid share and — for projects with a customer — billable meters and the charge at the plan price in the invoice currency (adds up to the estimate and invoice). CSV with Accept: text/csv.
• A standard key (cmr_…) now sees only its own usage in GET /v1/usage (including search and fetch); an account admin key (cma_…) sees the whole customer account.
• Changed for customer accounts: cost_usd (provider spend) and budget are no longer returned for projects of an external customer — charges at the plan price replace them. Projects without a customer keep both. Latency percentiles are now estimated from a histogram.
• New GET /v1/keys and GET /v1/keys/self: the keys visible to the caller, by prefix, label, role and status.
• group_by=hour (last 35 days), GET /v1/usage/timeseries (hour or day buckets, zero-filled, for charts) and GET /v1/usage/domains (busiest fetched domains per key, 90 days).
• GET /v1/requests (request ledger: cursor pagination, filters, charge per request, CSV) and GET /v1/requests/{request_id}.
• Per-key limits set by the operator (monthly spend cap in the invoice currency, monthly request quotas per category, requests per second, allowed categories), enforced with 403 key_category_not_allowed, 429 key_quota_exhausted, 429 key_budget_exhausted and 429 key_rate_limited. GET /v1/quota shows every limit with what is used, left and projected; /v1/keys show each key's limits. New headers on every authenticated response: RateLimit-Policy, RateLimit, and for keys with limits X-Cormorant-Key-Budget-Remaining and X-Cormorant-Quota-Remaining.
• Usage alerts: POST/GET/DELETE /v1/alerts register HMAC-SHA256-signed webhooks for 50 / 80 / 100 % of a key's, project's or the account's limits and allowances (retried with backoff); GET /v1/quota lists the month's events. The operator can also have alerts emailed.
• New guide Usage, spending and quotas (curl, Python and CSV examples), and the MCP tool cormorant_usage (usage summary, quota, latest requests, top domains in compact form for agents).

Charges, not provider costs#

• Changed for customer accounts: cost_usd in search and fetch results (sync and batch items) and cost_usd/cost_ok_usd in GET /v1/account usage are null for keys of an external customer. What you are charged is in GET /v1/usage and GET /v1/requests (charge). Projects without a customer and internal projects are unchanged.
• New billing_category on every search and fetch result: the category (and so the plan meters) the request counts toward.
• Batch callbacks and alert webhooks connect to the address that passed the SSRF check (no DNS rebinding).

Dead domains, archived copies for missing pages#

• A host whose name does not exist (NXDOMAIN twice within 6 hours) answers error: "domain_dead" at once for 6 hours, without trying any tier. Temporary DNS failures are unaffected (dns_error). New attempt reasons dns_nxdomain and domain_dead (pseudo-tier dns).
• With allow_archive (the default), a fetch that ends in domain_dead or http_404/http_410 returns a recent Wayback snapshot instead when one exists: tier: "archive", archived_at and the new field served_from_archive_because (domain_dead or not_found). Without a snapshot, alternative sources are looked for (as for a 404).

2026-10-19#

Account admin keys and additional keys#

API keys now have a role, fixed at creation and visible in the prefix: account admin keys start with cma_, additional (standard) keys with cmr_ as before. Every authenticated /v1 response carries X-Cormorant-Key-Role: account_admin|standard. Existing keys are standard keys.

Large PDFs in batch fetches#

• Batch fetch items that leave max_bytes unset accept PDFs up to 200 MB (previously too_large above 15 MB); an explicit max_bytes applies to PDFs too, and sync /v1/fetch keeps max_bytes. The exact bytes are available from body_url. Such a download may take at most 5 minutes (error: timeout, attempt reason pdf_download_timeout).
• New response fields pdf_page_count and text_truncated: PDF text extraction stops after 5,000 pages or 5,000,000 characters.

Service status#

• GET /v1/status: overall and per-component status (search, fetch, browser rendering, archive, paid tiers, batch jobs, storage) with reason codes, aggregate metrics, incidents and maintenance — no API key needed. Also GET /v1/status/summary, GET /v1/status/components/{id} and GET /v1/status/history. Status page at /status; see the monitoring guide.

Article metadata#

• Fetch results (sync, cache hits and batch items) have a new meta object for HTML pages: author, published_at, modified_at, language, canonical_url, publisher, site_name, description, section, word_count — from JSON-LD, OpenGraph, citation_* meta tags and the HTML. Pages cached before this change answer meta: null until they are fetched again.

Developer portal#

• Documentation at /docs: guides, an interactive API reference and the API contract; agent-readable /llms.txt and /llms-full.txt; every guide as Markdown at /docs/<page>.md.
• The OpenAPI document (/openapi.json) now describes every field, endpoint and error response, declares Bearer authentication and documents the batch completion callback as a webhook. Operation ids: search, fetch, batchSearch, batchFetch, getJob, cancelJob, getUsage, getBody, headBody.
• An MCP server exposes Cormorant as tools for AI agents.

Accounts, plans and invoices#

Projects can belong to a customer account with a price plan. New endpoints: GET /v1/account (project and customer usage by billing category, plan, current-month estimate), GET /v1/account/invoices, GET /v1/account/invoices/{number} and …/pdf. Keys of a suspended account get 403 customer_suspended on every /v1 endpoint.

Unavailable domains, honest empty searches#

• A domain where every tier failed in 3 consecutive fetches is skipped for a while: fetches answer at once with error: "domain_unavailable" and the new field retry_after_s.
• Search results carry unresponsive_engines. A search that the engines answered but that found nothing is results: [] with error: null.
• Learned tiers now expire after 3 days as documented.

Web Render retries and difficulty#

Busy Web Render answers are retried once (two attempts), and the difficulty that worked for a domain is remembered. Web Render attempts carry detail with the difficulty used (low, medium, high).

Faster synchronous calls under batch load#

POST /v1/search and POST /v1/fetch run on dedicated queues and workers, so large batch jobs no longer push synchronous calls into timeouts. No API change.

Per-request fetch controls#

New fetch fields: allow_paid (never use the paid tiers), tiers (allowlist of tiers), timeout_s (deadline for the whole fetch, 5–150 s), country (geotargeting for paid tiers), user_agent (must be on the project's allowlist, else 403 user_agent_not_allowed) and accept_language. New fetch errors: timeout, no_tier_allowed.

Cache hits honour the request#

A cached page is used only when it satisfies the request (render, min_text_chars, max_bytes, tiers); otherwise the page is fetched again. Best-effort pages with thin text are no longer cached. include_html is honoured for synchronous fetches too (default true there, false in batches).

Provenance#

Fetch results carry redirects (each hop), headers (an allowlist of the origin's response headers) and origin_content_type. final_url is null when it is not known (Web Render) instead of guessed; archived copies report the Wayback Machine URL.

Non-HTML content#

Fetch results carry content_kind (html, pdf, text, json, xml, binary). JSON and XML documents return useful text; binary documents return their bytes via body_url.

Exact retrieved bytes#

Every fetch stores the exact bytes it retrieved. Results carry body_url and body_kind (origin or rendered_dom); GET /v1/bodies/{sha256} streams them to projects that retrieved them.

Domain-aware batch admission#

Batch fetch items are admitted per domain, so a domain that dominates a job drains at its polite rate while other domains proceed. No API change.

2026-10-18#

v1#

Search with SearXNG first and a Brave fallback, fetch with tier escalation (direct, browser, archive, residential proxy, Web Render), shared caches, batch jobs with idempotency keys and callbacks, per-project keys, rate limits, budgets and GET /v1/usage. Job results are kept 30 days.