A Small TOON Parser in Python (Tested on All 185 Datasets)

TOON is small enough that a useful parser fits on one screen. This one is 83 lines of standard-library Python. It reads the TOON that this site publishes and returns the original data: we run it over all 185 .toon files in the catalog and compare the result with the matching .json. All 185 match exactly.

What it handles

• Objects (key: value, nested by two-space indentation).
• Inline arrays of primitives: tags[3]: a,b,c.
• Tabular arrays: rows[2]{id,name}: followed by one comma-separated row per record.
• List items for everything else (- value, - key: value, nested arrays).
• Quoted strings with escapes, plus true, false, null, and numbers.

The parser

import json, re

KEY = r'[A-Za-z_][A-Za-z0-9_.]*|"(?:[^"\\]|\\.)*"'
HEADER = re.compile(rf'^({KEY})?\[(\d+)\](?:\{{([^}}]*)\}})?:(?: (.*))?$')
FIELD = re.compile(rf'^({KEY}):(?: (.*))?$')
NUMBER = re.compile(r'^-?\d+(\.\d+)?([eE][+-]?\d+)?$')

def unquote(tok):
    return json.loads(tok) if tok.startswith('"') else tok

def scalar(tok):
    if tok.startswith('"'):
        return json.loads(tok)
    if tok in ('true', 'false', 'null'):
        return {'true': True, 'false': False, 'null': None}[tok]
    if NUMBER.match(tok):
        return float(tok) if re.search(r'[.eE]', tok) else int(tok)
    return tok

def cells(line):
    """Split on commas that are outside double quotes."""
    out, buf, quoted, esc = [], '', False, False
    for ch in line:
        if quoted:
            buf += ch
            if esc: esc = False
            elif ch == '\\': esc = True
            elif ch == '"': quoted = False
        elif ch == '"': quoted = True; buf += ch
        elif ch == ',': out.append(buf); buf = ''
        else: buf += ch
    return out + [buf]

def parse(text):
    lines = [((len(l) - len(l.lstrip(' '))) // 2, l.strip()) for l in text.split('\n') if l.strip()]
    pos = 0

    def array(m, depth):
        nonlocal pos
        pos += 1
        n, fields, inline = int(m.group(2)), m.group(3), m.group(4)
        if inline: return [scalar(c) for c in cells(inline)]
        if n == 0: return []
        if fields is not None:
            names = [unquote(f) for f in cells(fields)]
            rows = []
            for _ in range(n):
                rows.append(dict(zip(names, (scalar(c) for c in cells(lines[pos][1])))))
                pos += 1
            return rows
        items = []
        for _ in range(n):
            content = lines[pos][1]
            if content == '-':
                items.append({}); pos += 1
            else:
                lines[pos] = (depth + 2, content[2:])        # hyphen line becomes the first field
                head = HEADER.match(content[2:])
                if head and head.group(1) is None: items.append(array(head, depth + 1))
                elif FIELD.match(content[2:]): items.append(obj(depth + 2))
                else: items.append(scalar(content[2:])); pos += 1
        return items

    def obj(depth):
        nonlocal pos
        out = {}
        while pos < len(lines) and lines[pos][0] == depth:
            content = lines[pos][1]
            m = HEADER.match(content)
            if m:
                out[unquote(m.group(1))] = array(m, depth)
                continue
            key, rest = FIELD.match(content).groups()
            pos += 1
            if rest is None:
                out[unquote(key)] = obj(depth + 1) if pos < len(lines) and lines[pos][0] > depth else {}
            else:
                out[unquote(key)] = scalar(rest)
        return out

    head = HEADER.match(lines[0][1])
    if head and head.group(1) is None: return array(head, 0)
    return obj(0)

How it works

Each line becomes an indentation depth and its trimmed text. A header line (key[N]…:) tells the parser exactly how many items follow, so there is no guessing where an array ends. Tabular rows are zipped with the field names from the header. List items that are objects put their first field on the hyphen line, so the parser rewrites that line as a normal field one level deeper and reuses the object reader.

What it skips

Nested field groups in headers (pos{x,y}), alternative delimiters, and strict-mode validation. The published TOON format has all three; none are needed to read this catalog. For production use, take a maintained implementation: the reference library and its specification cover every case.

Test it yourself

Fetch any dataset in both formats and compare: /static-data/dataset/{slug}.toon and /static-data/dataset/{slug}.json. parse(toon_text) == json.loads(json_text) should hold. Before extending the parser, add tests for the empty array (key: []), a string that looks like a number ("05"), and a quoted comma inside a tabular cell. Those three catch most mistakes.