Why a Catalog Beats One Big Data Dump

The fastest way to publish reference data is to concatenate everything into one file and upload it. It takes an afternoon. It also creates problems that show up months later, usually on the day something needs to change.

What a single dump costs you

A combined file has four properties that each look harmless in isolation:

• One URL, so one cache key. Change a single row and every consumer's cache is invalid.
• One download size. Everyone pays for all 185 datasets to read three of them.
• One schema. A field name chosen for the biology data now applies to the law data whether it fits or not.
• One page. There is nothing for a search engine to rank and nothing for a person to link to. The whole thing is one opaque download.

What a catalog gives you instead

Splitting the same data into one document per dataset is barely more work, and it changes all four properties at once:

• A stable URL per dataset. /static-data/dataset/llm-glossary.json is a cache key, a citation target, and a thing you can paste into a ticket.
• Proportional cost. A consumer fetches the glossary and nothing else. Across the whole catalog that is 185 small files instead of one large one, and most clients touch a handful.
• Per-dataset shape. Each dataset carries its own fields under a shared envelope — the designing datasets for consumers note covers what belongs in the envelope and what does not.
• Something to link to. Each dataset gets a readable page, so people can cite it and crawlers can index it. This is why the catalog exists as HTML alongside the raw files.

You do not have to choose

The usual objection is that per-dataset files make bulk access painful. They do not, if you also publish an index. This site ships both:

• A single combined index of every category and dataset summary — 57,257 bytes as JSON, 34,554 as TOON — for anything that needs the whole map.
• One file per dataset for anything that needs the data.

The index is cheap to re-fetch and changes when the catalog changes. The per-dataset files change only when their own content changes, which is what makes them cacheable. That split is the reason static files and the API can coexist without duplicating effort.

The migration nobody plans for

Here is the practical test. Ask what happens when one dataset gets a correction. With a single dump, you re-publish the dump, every consumer re-downloads it, and nothing tells anyone which rows changed. With per-dataset files, exactly one file's bytes change, one cache entry invalidates, and consumers that never touch that dataset are unaffected.

You can go further and publish a changelog or a version per dataset. Whether you do depends on how many consumers you have — the trade-offs are covered in versioning without breaking consumers.

When a single dump is the right answer

Sometimes it is. If the data is genuinely one table, if it is small, and if every consumer needs all of it, splitting adds indirection for no benefit. A 40 KB CSV of country codes does not need a catalog.

The signal to watch is whether consumers ask for different things. The moment someone wants only the networking rows, you have a catalog whether or not you planned one.

How this one is built

The storage is a single database and a single source of truth; only the serving is split. A build step reads each dataset and writes its three encodings plus a readable page, so the pieces cannot drift from each other. If you are publishing your own data, the minimal SEO checklist covers the publishing side, and stable identifiers covers the naming decision you will live with longest.