The 30 Lines I Deleted by Changing One Thing

The bug showed up in production. A chatbot built on Claude Haiku was occasionally outputting conversational text before the JSON block—something like “Here’s the information you requested:” followed by the JSON the frontend expected. The frontend parsed the raw message, so that prefix appeared in the chat bubble as garbage metadata.

The fix attempts followed the obvious path: tell the model more firmly not to do it. “CRITICAL: only output JSON. No preamble.” Each variation worked until it didn’t. Haiku would comply for a while, then slip.

---

What I Tried

Switched to Anthropic tool use. Instead of asking the model to output JSON in a specific format, define a tool with a schema and set tool_choice to force the model to call it. The model has to populate the schema—it can’t output anything else.

Same model. Same cost. Different contract.

Before: a response parser that tried to extract JSON from whatever the model returned. A regex fallback. An extractJson() function that handled the prefix-text case, the markdown code block case, the stray-newline-before-the-brace case. About 30 lines.

After: none of that code. The tool schema defines the shape. The model fills it in.

---

What Actually Happened

Zero parsing code needed. The structured data comes back in the tool_use response block, not in the text content—so there’s nothing to extract, nothing to fall back on, and nothing to fail.

The reframe that helped: I was treating the model as a text generator that I needed to constrain with instructions. Tool use treats it as a function caller. Different abstraction, and the right one for this use case.

---

The Part Worth Keeping

Prompt-based JSON extraction works until it doesn’t, and “until it doesn’t” happens unpredictably. Tool use moves structure enforcement out of your parsing code and into the API contract.

If you’re maintaining regex fallbacks to handle model output variance, that’s the signal. The fix isn’t a better prompt.