Sync vs batch

Cormorant offers the same work two ways: synchronous calls that wait for the answer, and batch jobs that queue many items and let you collect the results later. Both run exactly the same search and fetch logic and return the same result objects.

Which one?#

Rule of thumb: more than about 20 items, or anything that does not need an answer right now, goes into a batch. Synchronous calls hold a connection open for the whole wait; a batch of 5,000 fetches is one HTTP request and drains at the fastest polite rate for each domain.

Synchronous calls#

A sync call returns when the result is ready. Cache hits return in milliseconds. When the result is not ready in time you get

{"detail": {"error": "timeout", "request_id": "req_6b1f2c0e9a7d4b5c8e3f1a2b3c4d5e6f"}}

with status 504. The work usually still completes in the background and lands in the shared cache, so retrying the same request a little later is often instant. Set timeout_s on fetches to bound the wait yourself.

Batch jobs#

1. Create the job#

curl -sS https://api.cormorant.ch/v1/batch/fetch \
  -H "Authorization: Bearer $CORMORANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "idempotency_key": "chipiron-run-2026-10-20-fetch-0001",
        "callback_url": "https://chipiron.example/hooks/cormorant",
        "items": [
          {"url": "https://www.holcim.com/investors"},
          {"url": "https://www.example.com/annual-report-2025.pdf", "allow_paid": false}
        ]
      }'

import os

import httpx

client = httpx.Client(
    base_url="https://api.cormorant.ch",
    headers={"Authorization": f"Bearer {os.environ['CORMORANT_API_KEY']}"},
    timeout=60,
)

urls = ["https://www.holcim.com/investors", "https://www.example.com/annual-report-2025.pdf"]
created = client.post(
    "/v1/batch/fetch",
    json={
        "idempotency_key": "chipiron-run-2026-10-20-fetch-0001",
        "items": [{"url": url} for url in urls],
    },
)
created.raise_for_status()  # 202 new job, 200 existing job with the same key
job_id = created.json()["job_id"]

const BASE = "https://api.cormorant.ch";
const headers = {
  Authorization: `Bearer ${process.env.CORMORANT_API_KEY}`,
  "Content-Type": "application/json",
};

const urls = ["https://www.holcim.com/investors", "https://www.example.com/annual-report-2025.pdf"];
const created = await fetch(`${BASE}/v1/batch/fetch`, {
  method: "POST",
  headers,
  body: JSON.stringify({
    idempotency_key: "chipiron-run-2026-10-20-fetch-0001",
    items: urls.map((url) => ({ url })),
  }),
});
if (!created.ok) throw new Error(`${created.status}: ${await created.text()}`);
const { job_id } = await created.json();

Each item takes the same fields as the single endpoint (POST /v1/search or POST /v1/fetch), so you can mix options per item. Limits: 50,000 items per job (413 above that) and a request body of at most 64 MB — split larger runs into several jobs.

2. Wait: poll or callback#

Poll GET /v1/jobs/{job_id} with a gentle interval (every 5–30 s, longer for big jobs). The job moves queued → running → completed (or cancelled), and counts tells you how far it is:

{
  "job_id": "1d0c5b8e-6f7a-4c1b-9e2d-3a4b5c6d7e8f",
  "kind": "fetch",
  "status": "running",
  "counts": {"total": 3000, "pending": 1200, "queued": 380, "done": 1390, "failed": 30, "cancelled": 0},
  "created_at": "2026-10-19T08:00:00+00:00",
  "updated_at": "2026-10-19T08:05:12+00:00",
  "finished_at": null,
  "callback_url": null,
  "callback_status": null,
  "items_url": "/v1/jobs/1d0c5b8e-6f7a-4c1b-9e2d-3a4b5c6d7e8f?include=items&offset=0&limit=100"
}

Item states: pending (waiting for admission), queued (admitted, running soon or now), done, failed (finished with an error; the result is kept), cancelled.

Or pass callback_url when creating the job. When the job finishes, Cormorant sends one POST to it:

{"job_id": "1d0c5b8e-6f7a-4c1b-9e2d-3a4b5c6d7e8f", "kind": "fetch", "status": "completed",
 "total": 3000, "succeeded": 2970, "failed": 30, "finished_at": "2026-10-19T09:41:02+00:00"}

Any 2xx answer counts as delivered; Cormorant tries 3 times and records the outcome in the job's callback_status. The callback is not signed: treat it as a hint, then read the job through the API with your key. Keep polling (slowly) as a fallback in case your endpoint was down. Callback URLs must be public http(s) addresses.

3. Page through the results#

Add include=items and page with offset/limit (up to 1,000 per page). Filter with status=failed to look at failures only.

import time


def wait(job_id: str) -> dict:
    delay = 5.0
    while True:
        job = client.get(f"/v1/jobs/{job_id}").json()
        if job["status"] in ("completed", "cancelled"):
            return job
        time.sleep(delay)
        delay = min(delay * 1.5, 60)


def items(job_id: str, status: str | None = None):
    offset = 0
    while True:
        params = {"include": "items", "offset": offset, "limit": 1000}
        if status:
            params["status"] = status
        page = client.get(f"/v1/jobs/{job_id}", params=params).json()
        yield from page["items"]
        if len(page["items"]) < 1000:
            return
        offset += 1000


job = wait(job_id)
for item in items(job_id):
    result = item["result"] or {}
    print(item["index"], item["status"], item["error"], len(result.get("text") or ""))

async function wait(jobId: string) {
  let delay = 5_000;
  for (;;) {
    const job = await (await fetch(`${BASE}/v1/jobs/${jobId}`, { headers })).json();
    if (job.status === "completed" || job.status === "cancelled") return job;
    await new Promise((r) => setTimeout(r, delay));
    delay = Math.min(delay * 1.5, 60_000);
  }
}

async function* items(jobId: string, status?: string) {
  for (let offset = 0; ; offset += 1000) {
    const params = new URLSearchParams({ include: "items", offset: String(offset), limit: "1000" });
    if (status) params.set("status", status);
    const page = await (await fetch(`${BASE}/v1/jobs/${jobId}?${params}`, { headers })).json();
    yield* page.items;
    if (page.items.length < 1000) return;
  }
}

await wait(job_id);
for await (const item of items(job_id)) {
  console.log(item.index, item.status, item.error, item.result?.text?.length ?? 0);
}

Each item has its index in your original list, its input, the result (the same object POST /v1/fetch or POST /v1/search returns, with request_id <job_id>:<index>) and an error. result is null with error: "result_unavailable" when the stored result cannot be read right now — read that page again later.

Idempotency#

idempotency_key is required and scoped to your project. Posting the same key again returns the existing job (200, created: false) instead of creating a second one, so a client that crashed or timed out while creating a job can simply retry. Cormorant recognises a replay by key, kind and item count: the same key with a different kind or number of items is 409. A changed list with the same length is not detected — derive keys from your run's identity (for example <project>-<run-id>-<chunk-number>, or a hash of the item list) and never reuse one for different content.

Cancelling#

DELETE /v1/jobs/{job_id} stops admitting new items within a few seconds. Pending items become cancelled, items already queued skip themselves, and finished results stay readable. The response is the job status.

Retention#

Jobs and their item results are kept for 30 days after the job finishes, then deleted. Copy the results you need before that. The exact bytes behind fetch results (body_url) follow their own retention (see Fetching in depth).