TOON vs JSON: Token Savings Measured Across 12 Real Datasets

Marketing claims about token savings are easy. Real numbers are useful. We measured 12 production YJTOON datasets in three formats using a popular BPE tokenizer.

Method

• Same payload, three encoders: json_encode with no whitespace, yaml_emit with default style, and a strict TOON encoder.
• Tokens counted by a cl100k_base tokenizer (representative of modern chat models).
• Datasets ranged from 6 rows to 18,000 rows, mostly flat tabular reference data.

Results

Across the 12 datasets, TOON used between 38% and 61% fewer tokens than compact JSON. YAML landed in the middle, typically 10-15% better than compact JSON but 25-35% worse than TOON.

When TOON does not win

• Highly nested data with deep object trees (TOON tables shine on flat records).
• Datasets with many long string fields containing newlines or colons (escaping eats the savings).
• Round-trip workflows where the consumer needs to mutate the structure inside the model context.

When TOON wins big

• Lookup tables (status codes, country lists, RFC headers).
• Reference matrices (HTTP method x content-type).
• Anything you would otherwise paste into a system prompt as a fenced JSON block.

The full table is published as the toon-vs-json-tokens dataset on YJTOON, in all three formats.