Strip a large language model down to what it actually does and you are left with one operation: given a sequence of tokens, predict the next one. It does this very well and very fast, and nothing else. It cannot check today's exchange rate, run a query against your orders table, or send an email. It produces text, and text alone.
So how does an agent book a flight? Not directly. It writes down a precise instruction that says book a flight, and a piece of ordinary software you control reads that instruction and does the booking. The whole field of tool use rests on that handoff. And the handoff only works if the instruction the model writes is something a machine can parse without guessing. That second clause is where the real engineering lives.
Origin: how a text predictor learned to call an API
Tool use is a translation trick. The model still only predicts text. What changed is that we taught it to predict a very particular kind of text, a structured call, at the right moment, and we built a runtime that watches for it.
The mechanics are the same across providers. You describe your tools to the model as part of the request: each one gets a name, a plain-language description of what it does and when to use it, and a schema for its inputs written in JSON Schema. The model reads the user's message, decides a tool is needed, and instead of replying in prose it emits a structured block naming the tool and supplying arguments that fit the schema. Anthropic's documentation describes the cycle plainly: Claude returns a tool_use block, your code executes the operation, and you send the result back as a tool_result. OpenAI's function calling guide describes the identical loop with different field names. The model never runs anything. It requests. Your code disposes. The result then goes back into the context window as new text, and the model continues with that information in hand.
That last point dissolves a lot of confusion. A model that can run this loop, calling a tool, reading the result, calling another, is an agent. A model on a fixed script of calls you wrote is a workflow. That line is the single most useful distinction in applied AI, and it is covered in full in workflow versus agent.
How did the model learn to do this at all? It was trained to. The earliest influential demonstration was Toolformer, published by Meta AI and Universitat Pompeu Fabra in February 2023. Its method was self-supervised: let the model insert candidate API calls at plausible spots in ordinary text, execute them, and keep only the cases where the result made the surrounding text easier to predict. Train on those kept examples and the model learns when a tool helps. Toolformer was a fine-tuned GPT-J with 6.7 billion parameters, and it beat the 175-billion-parameter GPT-3 on math word problems, not because it was smarter but because it had learned to reach for a calculator. It handled only five fixed tools, a calculator, a calendar, a question-answering system, a Wikipedia search, and a translator; Berkeley's Gorilla project, released in mid-2023, pushed toward thousands of arbitrary, user-defined APIs. Today every frontier model ships with tool calling baked in through post-training, and Berkeley's Function Calling Leaderboard exists specifically to measure how well each one does it.
Present: why reliable structured output is the hard part
Here is the part the demos skip. Getting a model to call a tool is mostly solved. Getting it to emit a structured payload your code can parse, every time, across a million calls, is not.
The reason is baked into what the model is. It was trained on human text, and human text is conversational, hedged, and wrapped in politeness. A JSON parser is none of those things. It wants exactly {"city":"Berlin"} and nothing else. The model, left to its instincts, wants to be helpful. Those two wants collide, and the collision produces a catalog of failures anyone who has shipped this will recognize.
The model adds filler. You ask for a JSON object and it returns a valid one wrapped in a markdown code fence, preceded by "Sure, here is the JSON you requested" and followed by a friendly note about edge cases. The JSON is fine. The words around it crash json.loads(). The model renames fields: you asked for status and it decided current_state was clearer, so your code reaches for a key that is not there. One practitioner guide describes exactly this, an application crashing because the model renamed a field without warning. The model gets types wrong, returning {"score":"0.85"}, a string where your schema wanted a number. It emits invalid syntax outright: a trailing comma, an unescaped quote, JSONL where you wanted JSON. And the worst one is drift. A prompt that produced clean output for six months starts failing the week after a model update, because the new model's instincts are tuned slightly differently and your prompt was always a suggestion, never a guarantee.
The scale of the gap is measurable. OpenAI has reported that an older GPT-4, asked to follow an output schema with no enforcement, complied less than 40 percent of the time. That is the true baseline of prompt-only structured output, and it is why a request that "works" in a notebook is not the same as one that works in production. A system at 95 percent reliability still breaks thousands of times a day at volume, and each break is a malformed record or a user-visible error.
Present: the fix ladder
The field has converged on a ladder. Each rung is more reliable than the last, and you climb only as far as the task demands.
Rung one is prompting. You write "respond with only valid JSON, no other text" and you hope. It is free, it works perhaps 80 to 95 percent of the time for a simple shape, and it fails silently on the edge cases. A common companion here is a repair library. json_repair, a widely used Python tool, exists for one job: take the malformed JSON an LLM produced and fix the missing brace or trailing comma after the fact. That repair libraries are popular at all tells you how routinely the raw output is broken. Rung one is fine for a prototype, a poor foundation for anything that matters.
Rung two is function calling. Instead of asking for JSON in prose, you define a tool with a typed input schema and let the model fill it. This is a real step up: the schema gives the model a strong, structured target rather than a phrased hope, and reliability climbs into the high nineties. But function calling alone is still, in most classic implementations, best effort. The model is strongly steered toward your schema, not yet forced into it. It can still produce a valid type with a nonsense value, or wander when the schema is deep.
Rung three is the one that actually closes the gap: native constrained decoding, exposed by the providers as a strict schema mode. OpenAI calls it Structured Outputs and turns it on with strict: true. Google's Gemini API takes a response schema with the response MIME type set to JSON, and in November 2025 added full JSON Schema support through a responseJsonSchema field, so Pydantic and Zod models work directly. Anthropic offers strict tool use to guarantee a tool call matches its schema exactly. Same idea, three vendors. OpenAI's reported numbers show the jump: from that sub-40-percent prompt-only baseline to 100 percent schema compliance in their evaluations once strict mode is on. The output is not validated after the fact. It is made impossible to get wrong in the first place.
Present: what constrained decoding actually does
Constrained decoding sounds exotic. The idea is simple, and once you see it you cannot unsee it.
Recall the one operation a model performs: predict the next token. It does not pick a token outright. At every step it produces a probability score for every token in its vocabulary, tens of thousands of options, then samples from that distribution. Constrained decoding inserts one step between the scores and the sample. Before the model picks, a checker looks at the partial output so far and asks which of the candidate next tokens would keep the output structurally valid. Every token that would not is masked, its probability forced to zero. The model then samples only from what survives.
Make that concrete. The model has emitted {"score": and is about to choose its next token. Your schema says score is a number, so a grammar derived from that schema knows the only legal next characters are a digit, a minus sign, or whitespace. Every token starting with a letter, a quote, or a brace gets masked to zero. The model physically cannot emit "high" there, no matter how much its training pulls it that way. Walk that masking through the whole generation and the output is valid by construction, not by luck. Aidan Cooper's walkthrough puts it cleanly: the model's next-token predictions are constrained to only tokens that do not violate the required structure.
The machinery underneath is a grammar. A simple schema compiles to a finite state machine; a richer one, with nesting and recursion, compiles to a context-free grammar, the same class that describes a programming language's syntax. Academic work like grammar-constrained decoding formalizes this, and open tooling makes it usable: llama.cpp ships GBNF, a grammar format that forces local models to emit valid JSON or any custom syntax you write, and libraries like Outlines and Guidance do the same for the broader ecosystem.
Two honest caveats. First, constrained decoding needs access to the raw token probabilities, so it runs inside the model server. You get it through the provider's strict mode or by self-hosting; you cannot bolt it onto a plain API that only returns finished text. Second, valid is not the same as correct. The grammar guarantees the shape. It cannot guarantee the city is the right city or the number is the true number. Google's own documentation says this directly: structured output guarantees syntactically correct JSON, not semantically correct values. You still validate meaning in your own code. This is why Instructor, which pairs a schema with Pydantic validators and automatically retries when a value fails a semantic check, clears fourteen million downloads a month. Constrained decoding fixes the syntax problem. The meaning problem stays yours.
Future and impact: where tool use is heading
Two directions are clear.
The first is that strict structured output is becoming the default rather than the advanced option. The cost of getting it wrong is now well understood, and the providers have made the reliable path a single flag. Expect schema-enforced output to be the assumed baseline of any serious agent, with prompt-only JSON treated the way raw SQL string concatenation is treated today: fine in a demo, a liability in production. The practitioner guidance has already shifted to stop parsing strings and enforce a schema for every tool.
The second is standardization of the layer above the call. Once a model can reliably emit a structured call, the open question becomes how it discovers tools, learns their schemas, and reaches them across systems. That is the job of the Model Context Protocol, which standardizes the connection between a model and the tools and data it uses, the way a single port standardized how peripherals attach to a computer. Tool use is the primitive. MCP is the network it travels over, and it is covered in full in the Model Context Protocol explained.
There is a security edge worth naming. The moment a model can call tools, a malicious instruction hidden in a web page or document it reads can try to hijack those calls. Indirect prompt injection is an open problem, and one of its few genuine mitigations is structured output itself: a model constrained to emit only a typed, validated object has far less room for free-form exfiltration text than one writing open prose. The discipline that makes tool use reliable also makes it harder to abuse.
For a company putting agents into production, this is the unglamorous core of the work. The model that calls a tool is the easy 20 percent. The schema design, the strict mode, the validation layer, the regression tests that catch a model update before it reaches a customer, that is the 80 percent that decides whether an agent is a demo or a system. It is the seam where an implementation partner like Perform Digital earns its place: not choosing the model, but building the enforced, tested handoff between what the model says and what the software does.
The text predictor never stopped being a text predictor. We just got disciplined about the one kind of text that has to be exactly right.
Council summary
The post argues that tool use splits into two unequal halves: getting a model to decide a tool is needed, which post-training has largely solved, and getting it to emit a payload your code can parse every time, which it has not. It traces the arc from Toolformer teaching itself five APIs to today's strict schema modes, and its core lesson is the fix ladder: prompting, then function calling, then native constrained decoding, where a grammar masks every token that would break the structure before the model can sample it. The reader leaves knowing why prompt-only JSON failed roughly 60 percent of the time on the old GPT-4 baseline, why strict mode reaches full schema compliance, and the caveat that keeps engineers honest: a grammar guarantees valid shape, never correct meaning. For anyone shipping agents, the takeaway is to treat schema-enforced output as the default and keep owning semantic validation in your own code.
Comments