Lobsters Blogger: New Anthropic Models Fare Worse With Nonstandard Tools

The author of the blog pocoo.org shared an observation he found troubling: while trying to get to the bottom of tool-calling regressions in Anthropic's newest generation of models, he uncovered a problem that hadn't cropped up with older versions of the models.

The gist of the problem

By his account, the new models appear to have been heavily fine-tuned with reinforcement learning (RL) specifically for Anthropic's own Claude Code harness — a tool whose source code is closed. As a result, the models seem to be too precisely tailored to the tool-declarations format used in Claude Code.

If a developer describes a tool in a way that comes close to the Claude Code format but doesn't match it exactly, the model starts behaving unpredictably — function calls break. Previous generations of models showed no such defect: they handled similar but not identical tool descriptions without failures of this kind.

Why it matters

The author calls the finding both intriguing and problematic. What's at stake is a situation where improving a model along one dimension (namely, how well it performs in the vendor's own harness) can degrade its behavior in a broader context — that is, in third-party tools and integrations that don't fully replicate Anthropic's internal format.

The author published the post himself on Lobsters — a rare occasion, as he notes, when he decided to post his own material there rather than a link to someone else's. The publication sparked discussion in the community comments, though the source material doesn't go into the details of that discussion.

The author doesn't specify which models exactly are involved or which tools were used in the tests — the piece focuses on the mere fact of the regression and its likely cause, tied to over-fitting to the closed harness format.

Source: Lobsters