October 18, 2026 · 7 min read · Yunus Emre Vurgun
A reference dataset is clean when every record has the same shape, every value means what its field name says, and every duplicate is a decision someone made on purpose. Most of the work falls into four checks: shape, values, identity and labels. Run them as code on every build, because a dataset that passed last month can drift the first time someone edits it by hand.
This guide walks through those checks with the YJTOON catalog as a worked example. The counts come from a scan of the 205 published dataset files in static-data/dataset/ on 18 October 2026, so you can put the same questions to your own data.
null, a blank string and a missing key three different meanings, and enforce them.Before you read a single value, confirm that the structure is consistent. Two shape problems turn up most often.
The first is the container. In the catalog, 131 datasets keep their records in a top-level array, and 74 use a keyed object with one or more named lists inside. Both are valid JSON, and either can be the right design for a dataset. The trouble starts when one dataset is shaped inconsistently, or when an API makes consumers guess which shape they have. Choose one container per dataset and state it in the schema.
The second is the record. Within one list, rows should share the same keys, or the optional keys should be documented. Twenty-seven datasets have a record list whose rows use more than one key set. Some of that may be deliberate. The clearest case is the CAP theorem dataset, where seven rows sit in one list: five carry guarantee and implication, and two carry examples and tradeoff. A consumer written for the first pair will silently miss what the other two rows say. The check is simple. Group the records in each list by their sorted key names, and review every dataset that produces more than one group.
A field can look empty in three ways, and each one means something different. JSON has a null literal, defined in RFC 8259, but the standard does not say what null means for your data. A missing key usually means the field does not apply, or that nobody filled it in. A blank string is almost always a mistake, because a program reads it as a value while a person reads it as nothing. Pick a meaning for each, write it down, and enforce it.
The check on the catalog found no null values. It did find two blank strings in the also_known_as field of the behavioral design patterns dataset, and one value with leading or trailing whitespace in the notes field of the Go concurrency patterns dataset. Each is a small fix, and each is easy to miss when you read a handful of rows.
Units, date formats and character encodings fail in the same quiet way. The units, dates and encodings guide covers them in depth. Here the only requirement is that each one has a written rule in your schema.
Tags and other labels are free text, and free text drifts. The catalog has 611 distinct tags across its 205 datasets, and 499 of those tags appear on exactly one dataset. A long tail is normal, but it is also where spelling variants hide. Thirty tags contain a space and 132 contain a hyphen. Treat spaces and hyphens as equal and seven pairs of labels merge:
In each pair both spellings name the same thing, so a filter on one form misses the datasets tagged with the other. The fix is a rule, not a one-off cleanup. Choose the canonical form, lowercase and hyphen-separated like a slug, write one function that applies it, and run that function on every build. A label that does not survive the function should fail the build so the author sees it, rather than being rewritten without notice. The unicode normalization guide covers the related problems of composed characters and casing, and the stable identifiers guide explains why slugs are worth protecting.
Identical rows are the easy case. Serialize each record with its keys sorted, then count the results. No top-level array in the catalog contains two identical rows. The harder case is two rows that describe the same thing in different words. For that, decide the key first, such as a slug, a code or a name, and count rows per key. When a key appears twice you have three choices. Keep both if they describe different things and say so in a field. Delete one if it is a copy. Or merge them and record which value won. Whichever you choose, the decision belongs in the changelog, where the next editor can find it.
Optional metadata is where unevenness hides. In the catalog, 26 datasets carry provenance, reviewed_at, sources and reading, while 97 carry updated_at and 94 carry created_at. Uneven coverage is not a defect by itself. What matters is whether a reader can tell what an absent field means. "Not checked yet" and "not applicable" are different answers, and a consumer should not have to guess between them. If you publish sources, say whether a dataset without sources has been reviewed at all.
Scripts catch shape and spelling. They do not catch a record that is well formed and wrong. Before you publish, read a sample by hand. Use a fixed random seed so the sample can be reproduced, read enough records to see every shape in the data, then read a few more chosen at random. When a sampled record looks wrong, write the rule that would have caught it and add that rule to the script, so the next run checks it automatically.
A cleaning step that runs once is a snapshot. Make it a stage of the build instead: normalize labels, validate shapes, compare the output with the previous release, and stop when a check fails. The YJTOON build works this way. Its verification script walks the generated pages and fails on broken markup, dead internal links and sitemap entries that point at missing files, and the build stops when a required step fails. Those checks cover the website rather than the data, but they follow the same rule: a failing check blocks publication.
The comparison step matters most for a dataset other people rely on. A diff between the previous and the new output shows whether a cleaning rule changed more values than you meant to. If a change would break consumers, follow the approach in the versioning guide rather than changing values in place.
For the schema side of this list, start with the guide to validating a reference table with JSON Schema. The full list of published datasets is in the catalog, the same records are served by the API, and the raw file for one dataset is at /static-data/dataset/cap-theorem.json.
Delete only exact copies. If two rows share a key but differ, first decide whether they describe different things. If they do, keep both and add a field that tells them apart. If they describe the same thing, merge them and record which value you kept and why.
Use null when the field applies but no value is known, and leave the key out when the field does not apply. Avoid empty strings, because most programs treat them as real values.
A schema is cheap even for a small file. It states the rules in a form a validator can check, and it gives consumers something to test against. The JSON Schema primer is a good starting point.
There is no fixed number. Read enough records to see every shape in the data, then a few more chosen at random with a seed you record, so the sample can be reproduced.
For a published dataset, clean the source so that every consumer gets the same answer. Normalize at query time for input that users type, such as search terms, where the original text still matters.
Compare the old and new output, count the changed values for each field, and read a sample of the changes. A change you cannot explain is a defect until you have explained it.
_Sieve,_Flour_Sifter.JPG){kind=link}
