Writing ·
How to verify AI agent tool listings before trusting tool selection
Test supplier-authored tool claims against executed capability, then measure whether AI-agent selection delivers the requested result.
By Youssef Hemimy · MCP · agent tools · agent evaluation
Keep tool descriptions for discovery, but treat supplier-authored capability claims as hypotheses. Execute candidates on representative tasks, compare delivered outputs with advertised capabilities, and measure whether the agent selects a tool that actually completes the request. Winning selection is not proof of capability.
01
Supplier claim
Snapshot the listing and its advertised outputs.
02
Capability test
Execute representative requests; grade delivered fields and failures.
03
Buyer replay
Record candidate set, chosen tool, and accepted task result.
Selection share and task success are different measurements.
The MCP Tools specification includes human-readable descriptions in tool definitions. Anthropic's tool guidance recommends describing purpose, usage, and limitations. Accurate metadata helps selection; neither source says a vendor's description has been independently verified.
What the Toolflation study found
Tang et al. built two experimental shelves of executable Amazon and Instagram data tools, with ten tools and 360 target-field questions per shelf. They ran tools ahead of the market simulation to record which requests each could answer. Seller agents rewrote their own short listings to increase selection; buyers saw descriptions but not vendor identities, input schemas, usage counts, or price. The first tool choice was compared with the recorded capability matrix.
Across five seller models, listings gained field claims, including some that tool outputs failed to support on most tested targets. In seven of ten seller-model/platform combinations, the most capable tool lost at least ten percentage points of selection share. That did not always reduce buyer success: under four seller models, choices largely stayed with capable tools. In the most severe setting, reported buyer success fell from 89% to about 30% on Amazon and from 77% to 40% on Instagram. The paper distinguishes claims contradicted by a seller's own sample output from broader overclaims across targets.
These are outcomes in a constructed, single-call market—not estimates for live registries or complete agent workflows. The seller agents were instructed to increase selection, not to lie. Real buyers may use additional reputation, schema, or pricing signals unavailable in this experiment.
Check the listing against executable capability
The following is an operator test derived from the paper's capability matrix, not an intervention the authors validated in deployment.
| Check | Evidence to retain |
|---|---|
| Representative requests | Targets and requested fields, including missing fields, ambiguous targets, and execution failures. |
| Listing claims | Description, schema, supplier, revision, and each material capability claim. |
| Executed capability | Delivered output, field missing, target missing, and execution error graded separately. |
| Buyer outcome | Candidate set, order, selected tool, and accepted task result—not selection share alone. |
Run each candidate on the same authorized test panel. A field observed once is not an unconditional coverage promise. Mark claims supported, contradicted, or untested, and retest after descriptions or tool behavior change. Then replay representative buyer requests and compare chosen-tool share with accepted task success; the paper shows those measures can diverge.
The MCP specification warns that clients should treat annotations as untrusted unless supplied by trusted servers. That warning is specifically about annotations. Treating seller-authored descriptions as claims needing verification is a practical inference, not a quoted MCP requirement.
Keep selection and execution boundaries visible
This is not a prompt-injection finding. The useful decision is whether a registry has independent output evidence behind a persuasive listing. History-aware MCP routing helps make the router's candidate set observable; production agent tool contracts define execution outcomes; and custom MCP server production clarifies metadata and audit ownership. All three targets are published posts.
Sources
FAQ
Can an AI agent trust a tool description when selecting a tool?
Use descriptions for discovery, but treat supplier-authored capability claims as unverified until representative tool executions support them.
How should a tool registry check advertised capabilities?
Snapshot the listing and revision, run candidates on the same representative requests, grade delivered outputs separately from errors, then replay buyer selection and accepted outcomes.
Did the Toolflation study measure real tool marketplaces?
No. It tested constructed Amazon and Instagram data-tool markets with selected models. Its percentages cannot be generalized to live marketplaces.
Building something that has to hold up?
We do this work for teams — agent reliability hardening, custom MCP servers, and full-stack AI systems built to survive production.