# Using Cormorant from AI agents

Cormorant was built to be called by programs that read the web on someone's
behalf — including LLM agents. This page collects the patterns that keep an
agent fast, cheap, polite and within its context window.

## Three ways in

1. **MCP** — run the [Cormorant MCP server](/docs/mcp.md) and
   your agent gets `cormorant_search`, `cormorant_fetch`,
   `cormorant_batch_fetch`, `cormorant_job_status`, `cormorant_usage`
   (what it used, what it costs, what is left) and the company tools
   `cormorant_company_search`, `cormorant_company_get` and
   `cormorant_company_resolve` (register data with sources, see
   [Company data](/docs/companies.md)) as tools. Best for
   Claude Desktop, Claude Code and other MCP clients.
2. **HTTP tools** — describe `POST /v1/search` and `POST /v1/fetch` as tools
   in your framework. The [OpenAPI document](/openapi.json) has every field
   described and stable `operationId`s (`search`, `fetch`, `batchFetch`,
   `getJob`, …) for tool generators.
3. **Docs for the model** — [`/llms.txt`](/llms.txt) indexes these guides as
   Markdown; [`/llms-full.txt`](/llms-full.txt) is everything in one file,
   including a compact endpoint reference generated from the OpenAPI document.
   Every guide is also available as Markdown at `/docs/<page>.md`.

## Rules of thumb

| Situation | Do |
|---|---|
| Answering a question | Search, read the snippets, fetch the 1–3 best results. |
| More than ~20 URLs or queries | Use a batch job, poll gently, page the results. |
| Feeding text to the model | Send `include_html: false`; use `text`, trimmed to what you need. |
| PDFs | Use `pdf_text_pages` to pick and cite pages instead of the whole `text`. |
| Need the original file | Download `body_url` only when you must (spreadsheets, archives, exact quotes). |
| Unattended runs | `allow_paid: false` unless paying is part of the task; set `timeout_s`. |
| Errors | Retry only what is retryable; respect `Retry-After`; never hot-loop. |
| Creating jobs | Deterministic `idempotency_key`s, so a retried call never duplicates work. |
| Citing | Keep `final_url`, `retrieved_at` and `sha256` with every quote. |

## Pattern: research a question

```python
import os

import httpx

client = httpx.Client(
    base_url="https://api.cormorant.ch",
    headers={"Authorization": f"Bearer {os.environ['CORMORANT_API_KEY']}"},
    timeout=180,
)


def research(question: str, pages: int = 3, chars: int = 6000) -> list[dict]:
    found = client.post("/v1/search", json={"query": question, "max_results": 8}).json()
    sources = []
    for hit in found["results"][:pages]:
        page = client.post(
            "/v1/fetch",
            json={"url": hit["url"], "include_html": False, "allow_paid": False, "timeout_s": 45},
        ).json()
        if page["error"] is None and page["text"]:
            sources.append(
                {
                    "url": page["final_url"] or page["url"],
                    "title": page["title"] or hit["title"],
                    "retrieved_at": page["retrieved_at"],
                    "text": page["text"][:chars],
                }
            )
    return sources
```

```ts
const BASE = "https://api.cormorant.ch";
const headers = {
  Authorization: `Bearer ${process.env.CORMORANT_API_KEY}`,
  "Content-Type": "application/json",
};
const post = async (path: string, body: unknown) =>
  (await fetch(`${BASE}${path}`, { method: "POST", headers, body: JSON.stringify(body) })).json();

async function research(question: string, pages = 3, chars = 6000) {
  const found = await post("/v1/search", { query: question, max_results: 8 });
  const sources = [];
  for (const hit of found.results.slice(0, pages)) {
    const page = await post("/v1/fetch", {
      url: hit.url,
      include_html: false,
      allow_paid: false,
      timeout_s: 45,
    });
    if (page.error === null && page.text) {
      sources.push({
        url: page.final_url ?? page.url,
        title: page.title ?? hit.title,
        retrieved_at: page.retrieved_at,
        text: page.text.slice(0, chars),
      });
    }
  }
  return sources;
}
```

## Pattern: bulk enrichment

When an agent plans to read many pages, it should submit them all at once and
come back later rather than fetching one by one:

```python
import hashlib
import json
import time

urls = [...]  # hundreds of URLs
key = "agent-" + hashlib.sha256(json.dumps(sorted(urls)).encode()).hexdigest()[:32]
job = client.post(
    "/v1/batch/fetch",
    json={"idempotency_key": key, "items": [{"url": u, "allow_paid": False} for u in urls]},
).json()

job_id = job["job_id"]
while client.get(f"/v1/jobs/{job_id}").json()["status"] not in ("completed", "cancelled"):
    time.sleep(15)

page = client.get(f"/v1/jobs/{job_id}", params={"include": "items", "limit": 1000}).json()
texts = {item["input"]["url"]: (item["result"] or {}).get("text") for item in page["items"]}
```

Batches are rate-limited as one request, drain each domain at its polite rate,
retry transient failures themselves, and keep results for 30 days — a much
better fit for agents than hundreds of parallel synchronous calls. See
[Sync vs batch](/docs/sync-vs-batch.md) for paging beyond 1,000 items.

## Keeping token use low

- `include_html: false` on every fetch. HTML is often 5–20× larger than
  `text`.
- Trim `text` to a budget, or search it locally for the passages you need,
  before putting it into the prompt.
- Do not echo whole result objects into the context: `headers`, `attempts`,
  `redirects` and `pdf_text_pages` are for debugging and citations.
- Prefer search snippets to decide *whether* to fetch. Results carry every
  snippet each engine returned.
- Re-fetching within the cache lifetime is instant and free — no need for the
  agent to memorise pages it may need again.

## Being a good citizen

- Respect `robots_disallowed`; do not try to work around it.
- Handle `429` by waiting `Retry-After` seconds. On `504`, wait before
  retrying (the work usually finishes in the background and the retry becomes
  a cache hit).
- Treat `politeness_timeout` and `all_tiers_failed` as "not now", not "try
  again immediately". `domain_unavailable` comes with `retry_after_s`: do not
  ask for that domain again before then — use another source.
- Use a dedicated project (and key) per agent with a modest budget, so a
  runaway loop cannot spend more than intended. Check `GET /v1/usage`.

## A system prompt fragment

> You can search the web and read pages with the Cormorant tools. Search first
> and read the snippets; fetch only the most relevant 1–3 pages. Fetch with
> `include_html` false and use the returned `text`. If a fetch result has an
> `error`, do not retry it immediately; try another source. For more than 20
> URLs, use the batch tool and check the job status later. Cite `final_url`
> and `retrieved_at` for every fact you take from a page.