# Changelog

Changes to the Cormorant API, newest first. Within v1, fields are only ever
added — nothing is renamed or removed. The [API contract](/docs/api.md) is the
reference for every field mentioned here.

## Unreleased

### SEC EDGAR usage

SEC EDGAR requests (`/v1/sec/...` and fetches of SEC hosts) are metered as
kind `sec` in new billing categories `sec` and `sec_cached`
(`billing_category`, `GET /v1/usage?category=`, `GET /v1/requests?kind=sec`).
`GET /v1/usage` adds a `sec` block, `keys[]` adds `sec_requests`. SEC has no
provider cost and the standard plans do not price it. For key limits, SEC
requests count as `fetch`.

### SEC EDGAR endpoints

A new guide, [SEC EDGAR via Cormorant](/docs/sec-edgar.md), and MCP tools
`cormorant_sec_company`, `cormorant_sec_filings` and `cormorant_sec_document`.

New `/v1/sec/...` endpoints (see the [API contract](/docs/api.md#sec-edgar-v1sec)):
`GET /v1/sec/companies/resolve` (ticker, CIK or name → company),
`GET /v1/sec/companies/{cik}/submissions` (normalised filings, including the
older filing pages), `GET /v1/sec/companies/{cik}/filings/latest` (latest
10-K/20-F/40-F), `GET /v1/sec/filings/{cik}/{accession}` (documents of a
filing), `POST /v1/sec/documents/fetch` (a filing document, same answer as
`POST /v1/fetch`), `GET /v1/sec/companies/{cik}/facts` (XBRL company facts)
and `GET /v1/sec/search` (EDGAR full-text search). New errors `not_found`
and `invalid_identifier`.

### SEC EDGAR policy for fetches

Fetches of `www.sec.gov`, `data.sec.gov` and `efts.sec.gov` follow SEC's
fair-access rules automatically: Cormorant sends its own declared identity as
`User-Agent` (a client `user_agent`/`accept_language` is ignored for these
hosts — do not send your own), only the direct tier is used (never the
browser, archive, paid tiers or alternative sources), and one cluster-wide
rate limit (default 8 requests/s, at most SEC's 10) is shared by all clients.
New errors `rate_limited` and `sec_unavailable` (both with `retry_after_s`,
retried automatically in batches), `sec_not_configured` and
`redirect_not_allowed`. Filing documents are cached for a year,
`company_tickers` and submissions for 12 h (revalidated with conditional
requests), full-text search for 1 h. `/v1/status` lists a `sec` component
(`success_rate`, `throttle_wait_p95_ms`, `backoff_active`, `rate_limit_rps`).

### Company data

New product area: the company register (`company/v1`). See
[Company data](/docs/companies.md).

- `GET /v1/companies/search`, `GET /v1/companies/{id}` (by `cmp_…` or
  `scheme:value`, with `max_age`, `refresh`, `include`, `sources`),
  `GET /v1/companies/{id}/history` and the sub-resources `events`, `people`,
  `ownership`, `relationships`, `filings`, `financials`, `sources`.
- `POST /v1/companies/resolve`, batch jobs of kind `company`
  (`POST /v1/companies/batch`, `/batch/csv`, results also as
  `GET /v1/companies/batch/{job_id}.csv`).
- `GET /v1/companies/changes` and signed `company.changed` webhooks
  (`POST/GET/DELETE /v1/companies/watches`).
- `GET /v1/companies/sources`, `GET /v1/schemas/company` (JSON Schema).
- First source: GLEIF (LEI records, register-id crosswalk, parents).
- Metered as the new billing categories and meters `company` and
  `company_cached` (answered from the register without a source refresh);
  key limits and quotas accept the category `company`; `GET /v1/usage` adds
  a `company` block when there is company usage and `company_requests` per
  key.
- Status: the new group `company_data` lists one component per active
  source (`company_<source>`); components carry `group`.
- MCP: `cormorant_company_search`, `cormorant_company_get`,
  `cormorant_company_resolve`.

### North Data

- **Transport policy** for `northdata.com` / `northdata.de` (also for
  `POST /v1/fetch` of these hosts): fetched only through Massive — the
  residential proxy first (Chrome-consistent headers; a sticky session when a
  `session_key` is given, retried once with a fresh session after a proxy
  error), then Web Render when the proxy was blocked. Never from Cormorant's
  own egress (no `direct`, `browser`, `archive` or alternative sources);
  robots.txt is read through the proxy too. `allow_paid: false` (or a
  `tiers` list without a Massive tier) answers `error: paid_tier_required`,
  an exhausted budget `error: budget_exhausted`. A cluster-wide throttle for
  the whole site (default 2 in flight, 1 s between starts). Profile pages are
  served from the cache for 24 h, then revalidated (entries kept 7 days).
- Challenge detection also recognises AWS WAF, Sucuri and "unusual traffic"
  pages. A Massive `407` (credentials refused) pauses the residential tier
  cluster-wide for 5 minutes; `452` is reported as `massive_content_policy`.
- **Profile parser v3** (`parser_version: "3"`): parses HTML and the Markdown
  Massive Web Render sometimes returns (entity lists included); `legal_form`
  from the company's own name or legal-form block; sanitised JSON-LD names;
  section entities only from the section's own list/table (never footer or
  navigation links); entity links as on the page (`url`) and absolute
  (`absolute_url`); new optional fields `path_country`, `lei`, `industry`, and
  officer `role`/`since`/`until` when the page shows them. Network graph
  parsing and merging (graph companies → `related`, persons → `officers`).
- **New endpoints** `POST /v1/northdata/suggest`, `/discover`, `/profile` and
  `/graph` (authenticated, metered as fetches). Discovery: suggest first
  (name similarity ≥ `min_similarity`, default 70), else Cormorant's own web
  search (four query templates, URL score ≥ 0.35, early stop at 0.75).
  Profiles come with `fetch` provenance (`body_url`, `sha256`, `body_kind`,
  `tier`, `attempts`, `cost_usd`, `parser_version`) and, with
  `include_graph` (default), the merged network graph. Suggest and graph
  answers are cached 24 h. Errors are HTTP 200 with `error`: `not_found`,
  `blocked`, `rate_limited` (+ `retry_after_s`), `all_tiers_failed`,
  `parse_failed`, `invalid_identifier`, `paid_tier_required`,
  `budget_exhausted`, `robots_disallowed`, `timeout`.
- **Batch**: `POST /v1/batch/northdata` with items `{op: suggest|discover|profile|graph,
  …fields of that endpoint}`; results have the endpoint's shape. Items wait
  for the site throttle by going back to the queue, not by holding a worker.
- **Metering**: requests are recorded as kind `northdata` with their own
  billing categories `northdata` and `northdata_cached` (meters of the same
  names; unpriced = free unless a plan prices them). Their Massive traffic
  counts towards `paid_gb` (premium traffic), their bytes towards
  `delivered_gb`. `GET /v1/usage` adds a `northdata` block and
  `keys[].northdata_requests`; `category=northdata,northdata_cached` and
  `GET /v1/requests?kind=northdata` filter them; responses carry
  `billing_category`. Key limits count them as `fetch` (uncached ones also
  as `fetch_paid`). Mission Control shows them in the company data card (next to
  company and SEC requests, outside the fetch totals) and in the domain
  statistics.
- **Docs**: guide [North Data via Cormorant](/docs/northdata.md) (endpoints,
  `session_key`, throttling, caching, robots.txt, the field list and the
  differences to Chipiron's scraper); MCP tool `cormorant_northdata`.

### Account admin keys and additional keys

API keys now have a role, fixed at creation and visible in the prefix:
account admin keys start with `cma_`, additional (standard) keys with `cmr_`
as before. Every authenticated `/v1` response carries
`X-Cormorant-Key-Role: account_admin|standard`. Existing keys are standard keys.

### Usage per API key

Requests are attributed to the API key that made them (batch items to the key
that created the job). `GET /v1/usage?group_by=key` adds `keys` (usage per key
of the project), and `GET /v1/account` adds `keys` (the month's usage and
share of the charges per key of the account). Keys appear by `prefix` and
`label` only.

### Privacy fix

- **Removed** `massive_reported_account_wide` from `GET /v1/usage`: it was the
  paid provider's usage across all customers and is not customer data. This is
  the one exception to the add-only rule: it should never have been exposed.

### Cache revalidation

- New fetch field `revalidated`: `true` when an expired cache entry was
  confirmed unchanged by the origin (conditional GET answered `304`; attempt
  reason `not_modified`). The cached copy is returned with `cached: true`
  and no cost.
- Only cache entries that satisfy the request are revalidated, and new
  validators from a `304` are stored.
### Usage, charges and keys from the aggregates

- `GET /v1/usage` is rebuilt on the usage aggregates (refreshed every 5
  minutes; `as_of`, `final`, and `final` per day): `group_by` takes `key`,
  `project`, `category` and `day`, combinable; filters `key`, `project`,
  `category`; new `totals` and `groups` with failures by error, latency
  p50/p95, bytes, paid share and — for projects with a customer — `billable`
  meters and the `charge` at the plan price in the invoice currency (adds up
  to the estimate and invoice). CSV with `Accept: text/csv`.
- A standard key (`cmr_…`) now sees only its own usage in `GET /v1/usage`
  (including `search` and `fetch`); an account admin key (`cma_…`) sees the
  whole customer account.
- **Changed for customer accounts:** `cost_usd` (provider spend) and `budget`
  are no longer returned for projects of an external customer — charges at
  the plan price replace them. Projects without a customer keep both.
  Latency percentiles are now estimated from a histogram.
- New `GET /v1/keys` and `GET /v1/keys/self`: the keys visible to the caller,
  by prefix, label, role and status.
- `group_by=hour` (last 35 days), `GET /v1/usage/timeseries` (hour or day
  buckets, zero-filled, for charts) and `GET /v1/usage/domains` (busiest
  fetched domains per key, 90 days).
- `GET /v1/requests` (request ledger: cursor pagination, filters, charge per
  request, CSV) and `GET /v1/requests/{request_id}`.
- Per-key limits set by the operator (monthly spend cap in the invoice
  currency, monthly request quotas per category, requests per second, allowed
  categories), enforced with `403 key_category_not_allowed`, `429
  key_quota_exhausted`, `429 key_budget_exhausted` and `429 key_rate_limited`.
  `GET /v1/quota` shows every limit with what is used, left and projected;
  `/v1/keys` show each key's `limits`. New headers on every authenticated
  response: `RateLimit-Policy`, `RateLimit`, and for keys with limits
  `X-Cormorant-Key-Budget-Remaining` and `X-Cormorant-Quota-Remaining`.
- Usage alerts: `POST/GET/DELETE /v1/alerts` register HMAC-SHA256-signed
  webhooks for 50 / 80 / 100 % of a key's, project's or the account's limits
  and allowances (retried with backoff); `GET /v1/quota` lists the month's
  `events`. The operator can also have alerts emailed.
- New guide *Usage, spending and quotas* (curl, Python and CSV examples), and
  the MCP tool `cormorant_usage` (usage summary, quota, latest requests, top
  domains in compact form for agents).

### Charges, not provider costs

- **Changed for customer accounts:** `cost_usd` in search and fetch results
  (sync and batch items) and `cost_usd`/`cost_ok_usd` in `GET /v1/account`
  usage are `null` for keys of an external customer. What you are charged is
  in `GET /v1/usage` and `GET /v1/requests` (`charge`). Projects without a
  customer and internal projects are unchanged.
- New `billing_category` on every search and fetch result: the category (and
  so the plan meters) the request counts toward.
- Batch callbacks and alert webhooks connect to the address that passed the
  SSRF check (no DNS rebinding).

### Dead domains, archived copies for missing pages

- A host whose name does not exist (NXDOMAIN twice within 6 hours) answers
  `error: "domain_dead"` at once for 6 hours, without trying any tier.
  Temporary DNS failures are unaffected (`dns_error`). New attempt reasons
  `dns_nxdomain` and `domain_dead` (pseudo-tier `dns`).
- With `allow_archive` (the default), a fetch that ends in `domain_dead` or
  `http_404`/`http_410` returns a recent Wayback snapshot instead when one
  exists: `tier: "archive"`, `archived_at` and the new field
  `served_from_archive_because` (`domain_dead` or `not_found`). Without a
  snapshot, alternative sources are looked for (as for a 404).

## 2026-10-19



### Account admin keys and additional keys

API keys now have a role, fixed at creation and visible in the prefix:
account admin keys start with `cma_`, additional (standard) keys with `cmr_`
as before. Every authenticated `/v1` response carries
`X-Cormorant-Key-Role: account_admin|standard`. Existing keys are standard keys.

### Large PDFs in batch fetches

- Batch fetch items that leave `max_bytes` unset accept PDFs up to 200 MB
  (previously `too_large` above 15 MB); an explicit `max_bytes` applies to
  PDFs too, and sync `/v1/fetch` keeps `max_bytes`. The exact bytes are
  available from `body_url`. Such a download may take at most 5 minutes
  (`error: timeout`, attempt reason `pdf_download_timeout`).
- New response fields `pdf_page_count` and `text_truncated`: PDF text
  extraction stops after 5,000 pages or 5,000,000 characters.

### Service status

- `GET /v1/status`: overall and per-component status (search, fetch, browser
  rendering, archive, paid tiers, batch jobs, storage) with reason codes,
  aggregate metrics, incidents and maintenance — no API key needed. Also
  `GET /v1/status/summary`, `GET /v1/status/components/{id}` and
  `GET /v1/status/history`. Status page at [`/status`](/status); see the
  [monitoring guide](/docs/monitoring.md).

### Article metadata

- Fetch results (sync, cache hits and batch items) have a new `meta`
  object for HTML pages: `author`, `published_at`, `modified_at`,
  `language`, `canonical_url`, `publisher`, `site_name`, `description`,
  `section`, `word_count` — from JSON-LD, OpenGraph, `citation_*` meta tags
  and the HTML. Pages cached before this change answer `meta: null` until
  they are fetched again.

### Developer portal

- Documentation at `/docs`: guides, an interactive API reference and the API
  contract; agent-readable [`/llms.txt`](/llms.txt) and
  [`/llms-full.txt`](/llms-full.txt); every guide as Markdown at
  `/docs/<page>.md`.
- The OpenAPI document (`/openapi.json`) now describes every field, endpoint
  and error response, declares Bearer authentication and documents the batch
  completion callback as a webhook. Operation ids: `search`, `fetch`,
  `batchSearch`, `batchFetch`, `getJob`, `cancelJob`, `getUsage`, `getBody`,
  `headBody`.
- An [MCP server](/docs/mcp.md) exposes Cormorant as tools for AI
  agents.

### Accounts, plans and invoices

Projects can belong to a customer account with a price plan. New endpoints:
`GET /v1/account` (project and customer usage by billing category, plan,
current-month estimate), `GET /v1/account/invoices`,
`GET /v1/account/invoices/{number}` and `…/pdf`. Keys of a suspended account
get `403 customer_suspended` on every `/v1` endpoint.

### Unavailable domains, honest empty searches

- A domain where every tier failed in 3 consecutive fetches is skipped for a
  while: fetches answer at once with `error: "domain_unavailable"` and the new
  field `retry_after_s`.
- Search results carry `unresponsive_engines`. A search that the engines
  answered but that found nothing is `results: []` with `error: null`.
- Learned tiers now expire after 3 days as documented.

### Web Render retries and difficulty

Busy Web Render answers are retried once (two `attempts`), and the
difficulty that worked for a domain is remembered. Web Render attempts carry
`detail` with the difficulty used (`low`, `medium`, `high`).

### Faster synchronous calls under batch load

`POST /v1/search` and `POST /v1/fetch` run on dedicated queues and workers, so
large batch jobs no longer push synchronous calls into timeouts. No API change.

### Per-request fetch controls

New fetch fields: `allow_paid` (never use the paid tiers), `tiers` (allowlist
of tiers), `timeout_s` (deadline for the whole fetch, 5–150 s), `country`
(geotargeting for paid tiers), `user_agent` (must be on the project's
allowlist, else `403 user_agent_not_allowed`) and `accept_language`. New fetch
errors: `timeout`, `no_tier_allowed`.

### Cache hits honour the request

A cached page is used only when it satisfies the request (`render`,
`min_text_chars`, `max_bytes`, `tiers`); otherwise the page is fetched again.
Best-effort pages with thin text are no longer cached. `include_html` is
honoured for synchronous fetches too (default `true` there, `false` in
batches).

### Provenance

Fetch results carry `redirects` (each hop), `headers` (an allowlist of the
origin's response headers) and `origin_content_type`. `final_url` is `null`
when it is not known (Web Render) instead of guessed; archived copies report
the Wayback Machine URL.

### Non-HTML content

Fetch results carry `content_kind` (`html`, `pdf`, `text`, `json`, `xml`,
`binary`). JSON and XML documents return useful `text`; binary documents
return their bytes via `body_url`.

### Exact retrieved bytes

Every fetch stores the exact bytes it retrieved. Results carry `body_url` and
`body_kind` (`origin` or `rendered_dom`); `GET /v1/bodies/{sha256}` streams
them to projects that retrieved them.

### Domain-aware batch admission

Batch fetch items are admitted per domain, so a domain that dominates a job
drains at its polite rate while other domains proceed. No API change.

## 2026-10-18

### v1

Search with SearXNG first and a Brave fallback, fetch with tier escalation
(direct, browser, archive, residential proxy, Web Render), shared caches, batch
jobs with idempotency keys and callbacks, per-project keys, rate limits,
budgets and `GET /v1/usage`. Job results are kept 30 days.