Stop Feeding Agents Whole Datasets: The Lookup-First Pattern

Most agent demos start the same way: take a reference dataset, paste all of it into the system prompt, and let the model find what it needs. It works in the demo because the dataset is small and the question is easy. It stops working when the dataset grows, the question gets specific, or the request has to run a thousand times a day.

Why preloading feels right

Preloading has three genuine advantages. It removes a network call from the hot path. It makes behaviour deterministic, because the model always sees the same context. And it is trivially easy to implement — one string concatenation and you are done.

The cost is less obvious because it is paid per request rather than once. Every row the agent never reads is still a row the model has to attend to. A dataset with repeated keys on every row spends a meaningful share of its bytes on syntax instead of values, which is the problem the TOON comparison measures. And a prompt assembled at build time cannot answer a question about data that changed last week.

The lookup-first loop

The alternative keeps the data outside the context and fetches only what a task needs. The shape is four steps:

1. Classify the task into the small set of things your agent actually does. "Which port does this service use?" is a lookup, not a research task.
2. Fetch the smallest document that can answer it. One dataset file, not the whole catalog.
3. Select rows in code, not in the prompt. Filtering is a string comparison. Letting the model do it means paying for every row it rejects.
4. Put only the surviving rows in the prompt, with the source URL so the answer can be traced.

Step three is the one people skip, and it is the one that matters. A filter that runs in Python costs nothing per token.

What that looks like

The whole mechanism is a fetch, a filter, and a render. The interesting part is that the filter runs before the model sees anything:

import json, urllib.request

def fetch_rows(dataset, predicate, limit=8):
    url = f"https://yjtoon.com/static-data/dataset/{dataset}.json"
    with urllib.request.urlopen(url) as res:
        payload = json.load(res)

    rows = []
    for section in payload["data"].values():
        if not isinstance(section, list):
            continue
        for row in section:
            if predicate(row):
                rows.append(row)
    return rows[:limit]

Call it with a predicate that matches the task, and the prompt receives a handful of objects instead of the file. For a table like the network ports reference, a predicate on the service name is usually enough.

Keep a small index in context instead

You do not want the agent to guess which dataset to fetch. Give it a map, not the territory. Two options work well:

• A compact index of what exists. Category names, dataset titles, and one-line descriptions are enough for the model to choose a target. The combined catalog index is 34,554 bytes as TOON — small enough to ship once and re-fetch only when the catalog changes.
• Explicit routing. If your agent only does four things, map each one to a dataset in code and skip the model's judgement entirely. This is faster and removes a failure mode.

Either way, the index is a fixed cost that does not grow with the data. That is the property preloading cannot offer.

When preloading is still correct

Lookup-first is not a universal rule. Preload when the dataset is genuinely small (say, under a few thousand tokens), when every request needs most of it, or when the request is latency-critical and a network call is unacceptable. A twelve-row glossary belongs in the prompt. The LLM glossary dataset is a good example of something small enough to embed wholesale.

The signal that you have outgrown preloading is specific: you are editing the prompt to remove rows for a particular request. That edit is a filter, and it belongs in code.

Check that it actually helps

Two numbers tell you whether the change worked. The first is request bytes, which you can log exactly. The second is the size of the assembled prompt, which you can measure with your own tokenizer rather than guessing — the distinction between bytes and tokens is covered in bytes, tokens, and context windows.

Then check the answers did not get worse. Grounding each value back to the row it came from is the cheap way to do that, and the verification loop covers the mechanics. A lookup pipeline that returns wrong rows quickly is not an improvement.