Using Cormorant from AI agents

Cormorant was built to be called by programs that read the web on someone's behalf — including LLM agents. This page collects the patterns that keep an agent fast, cheap, polite and within its context window.

Three ways in#

1. MCP — run the Cormorant MCP server and your agent gets cormorant_search, cormorant_fetch, cormorant_batch_fetch, cormorant_job_status, cormorant_usage (what it used, what it costs, what is left) and the company tools cormorant_company_search, cormorant_company_get and cormorant_company_resolve (register data with sources, see Company data) as tools. Best for Claude Desktop, Claude Code and other MCP clients.
2. HTTP tools — describe POST /v1/search and POST /v1/fetch as tools in your framework. The OpenAPI document has every field described and stable operationIds (search, fetch, batchFetch, getJob, …) for tool generators.
3. Docs for the model — /llms.txt indexes these guides as Markdown; /llms-full.txt is everything in one file, including a compact endpoint reference generated from the OpenAPI document. Every guide is also available as Markdown at /docs/<page>.md.

Rules of thumb#

Pattern: research a question#

import os

import httpx

client = httpx.Client(
    base_url="https://api.cormorant.ch",
    headers={"Authorization": f"Bearer {os.environ['CORMORANT_API_KEY']}"},
    timeout=180,
)


def research(question: str, pages: int = 3, chars: int = 6000) -> list[dict]:
    found = client.post("/v1/search", json={"query": question, "max_results": 8}).json()
    sources = []
    for hit in found["results"][:pages]:
        page = client.post(
            "/v1/fetch",
            json={"url": hit["url"], "include_html": False, "allow_paid": False, "timeout_s": 45},
        ).json()
        if page["error"] is None and page["text"]:
            sources.append(
                {
                    "url": page["final_url"] or page["url"],
                    "title": page["title"] or hit["title"],
                    "retrieved_at": page["retrieved_at"],
                    "text": page["text"][:chars],
                }
            )
    return sources

const BASE = "https://api.cormorant.ch";
const headers = {
  Authorization: `Bearer ${process.env.CORMORANT_API_KEY}`,
  "Content-Type": "application/json",
};
const post = async (path: string, body: unknown) =>
  (await fetch(`${BASE}${path}`, { method: "POST", headers, body: JSON.stringify(body) })).json();

async function research(question: string, pages = 3, chars = 6000) {
  const found = await post("/v1/search", { query: question, max_results: 8 });
  const sources = [];
  for (const hit of found.results.slice(0, pages)) {
    const page = await post("/v1/fetch", {
      url: hit.url,
      include_html: false,
      allow_paid: false,
      timeout_s: 45,
    });
    if (page.error === null && page.text) {
      sources.push({
        url: page.final_url ?? page.url,
        title: page.title ?? hit.title,
        retrieved_at: page.retrieved_at,
        text: page.text.slice(0, chars),
      });
    }
  }
  return sources;
}

Pattern: bulk enrichment#

When an agent plans to read many pages, it should submit them all at once and come back later rather than fetching one by one:

import hashlib
import json
import time

urls = [...]  # hundreds of URLs
key = "agent-" + hashlib.sha256(json.dumps(sorted(urls)).encode()).hexdigest()[:32]
job = client.post(
    "/v1/batch/fetch",
    json={"idempotency_key": key, "items": [{"url": u, "allow_paid": False} for u in urls]},
).json()

job_id = job["job_id"]
while client.get(f"/v1/jobs/{job_id}").json()["status"] not in ("completed", "cancelled"):
    time.sleep(15)

page = client.get(f"/v1/jobs/{job_id}", params={"include": "items", "limit": 1000}).json()
texts = {item["input"]["url"]: (item["result"] or {}).get("text") for item in page["items"]}

Batches are rate-limited as one request, drain each domain at its polite rate, retry transient failures themselves, and keep results for 30 days — a much better fit for agents than hundreds of parallel synchronous calls. See Sync vs batch for paging beyond 1,000 items.

Keeping token use low#

• include_html: false on every fetch. HTML is often 5–20× larger than text.
• Trim text to a budget, or search it locally for the passages you need, before putting it into the prompt.
• Do not echo whole result objects into the context: headers, attempts, redirects and pdf_text_pages are for debugging and citations.
• Prefer search snippets to decide whether to fetch. Results carry every snippet each engine returned.
• Re-fetching within the cache lifetime is instant and free — no need for the agent to memorise pages it may need again.

Being a good citizen#

• Respect robots_disallowed; do not try to work around it.
• Handle 429 by waiting Retry-After seconds. On 504, wait before retrying (the work usually finishes in the background and the retry becomes a cache hit).
• Treat politeness_timeout and all_tiers_failed as "not now", not "try again immediately". domain_unavailable comes with retry_after_s: do not ask for that domain again before then — use another source.
• Use a dedicated project (and key) per agent with a modest budget, so a runaway loop cannot spend more than intended. Check GET /v1/usage.

A system prompt fragment#

You can search the web and read pages with the Cormorant tools. Search first and read the snippets; fetch only the most relevant 1–3 pages. Fetch with include_html false and use the returned text. If a fetch result has an error, do not retry it immediately; try another source. For more than 20 URLs, use the batch tool and check the job status later. Cite final_url and retrieved_at for every fact you take from a page.