Turn one page into a catalog

Some pages are worth owning: a reference table, a spec index, a “list of…” article you keep coming back to. You want it as data.

This is the loop we run ourselves. One Wikipedia article, List of stock characters, becomes a 470-entry catalog with 3,684 cited examples (a 438 KB JSON file) — one read, then parsing that is yours.

Below, the same job is written twice — once against the HTTP API, once from an AI agent over MCP — so you can compare them and pick.

The shape of it

The watch half of this page is withdrawn

This guide was written as build + watch: read the page once to build the catalog, then re-check it cheaply and rebuild only when it moved.

The watch half is gone for now. formats:["changeTracking"], formats:["sections"] and /v1/monitor all answer 410 monitoring-withdrawn while the extraction design is reworked. What remains below — read once, parse into records — works unchanged.

⚠️ Rebuilding on a schedule still works; it just costs a full read each time (1 credit) instead of a cheap check. There is no substitute format name.

The build half is one call:

	formats	you get back	costs
Build	["markdown"]	the full page body (data.data.body)	1 credit

One default to know before you trust a number: with no maxAge, a read may be answered from a stored capture up to 7 days old. That is the wrong setting while you are still writing the parser — send maxAge: 0 to force a live capture, and check provenance.track (store-cache versus fast-path) if you are unsure which you got. Same rule on the deep-read page.

Version A — HTTP API

Build. One read, then the parsing is yours:

import json, urllib.request

URL = "https://en.wikipedia.org/wiki/List_of_stock_characters"

def read(url: str, key: str) -> str:
    req = urllib.request.Request(
        "https://api.marrow.navii.online/v1/scrape",
        data=json.dumps({"url": url, "formats": ["markdown"]}).encode(),
        headers={
            "Authorization": f"Bearer {key}",
            "Content-Type": "application/json",
            # Name yourself. Our edge sits behind Cloudflare, which refuses
            # urllib's default `Python-urllib/3.x` with a 403 before the request
            # reaches us — and that 403 looks exactly like a bad key.
            "User-Agent": "my-app/1.0",
        },
    )
    with urllib.request.urlopen(req, timeout=180) as r:
        return json.load(r)["data"]["data"]["body"]   # note the two `data`s

That returns ~957,000 characters of markdown — the article’s table as | name | description | examples | rows (520 of them), with the site chrome and navigation boxes gone. What Marrow deliberately keeps is the link structure of the prose, so a wiki article still carries its footnote markers ([[12]](…) — 207 in this page) and its inline links; stripping those is your cleaner’s call, not ours. Both paths keep them, but not the same set: a deep read carries the links inside the prose it selected and leaves the surrounding chrome behind, so it returns fewer — on /wiki/Web_scraping, 346 links from the fast path against 288 from a deep read. A page whose links live in card grids rather than in prose can come back from a deep read with none of them; that is the extractor choosing prose, not a format difference. Turning rows into records is ordinary parsing, but one detail cost us a re-run: do not split rows on blank lines. Some cells contain a blank line (two-paragraph descriptions), and splitting there silently drops those rows’ examples. Accumulate from | until three cells have closed instead.

Version B — from an AI agent (MCP)

The same call, spoken instead of typed. With @marrowdev/cloud-mcp installed, the agent has marrow_read:

{ "tool": "marrow_read",
  "arguments": { "url": "https://en.wikipedia.org/wiki/List_of_stock_characters",
                 "formats": ["markdown"] } }

⚠️ The difference that matters is where the page lands. A 957 KB body returned to an agent is 957 KB of context. The escape hatch used to be reading it as ["sections"] with spool: true, so the chunks landed in a local file and the model never held the page — that is withdrawn with the rest. Until it returns, prefer version A for anything this large, and use the agent only to explore a source before you commit a parser to it.

Which one

	HTTP API	MCP agent
Extraction logic	you write it, it is deterministic and testable	the model writes/adapts it — good when the layout drifts
The 957 KB body	goes straight to your parser	lands in your context — there is no spool today
Best at	a catalog you rebuild for years	a page whose shape you do not know yet
Scheduling	cron / systemd	whatever wakes your agent

We use A for the catalog itself and B for exploring a new source before committing a parser to it.

Things that will bite you

• Do not send max_tokens when you are building a catalog. Truncation works — including on cache hits — so a budget silently gives you a shorter page and a shorter catalog.
• Do not split rows on blank lines. Some cells contain a blank line (two-paragraph descriptions), and splitting there silently drops those rows’ examples. Accumulate from | until three cells have closed instead.
• Pin maxAge while you are writing the parser. A 7-day-old capture makes a parser bug and a source change look identical.

Cost

Every read above is 1 credit, measured on the header the API returns (x-marrow-credits-remaining). ⚠️ Without the cheap watch, a daily rebuild is a full read a day — ~30 credits a month, and it re-reads on the days nothing moved. Free plan is 500 credits/month, so a daily rebuild still fits — see pricing.