How to filter agent policies without missing critical rules

Put a cheap relevance filter before expensive agent-policy evaluation only if you can measure the policies it misses. Build activating examples and hard near-misses for each rule, tune for recall on held-out conversations, include uncertain rules, and keep authorization outside the filter. The benefit is fewer policies sent downstream; the failure mode is that an excluded rule cannot govern the answer.

Separate discovery from enforcement

An agent with dozens of behavioral rules does not need every rule in every prompt. But relevance and authorization are different decisions. A discovery layer may choose which policy text to show the model. The action boundary still has to enforce the current policy, including rules that depend on identity, state, or other rules. Treat the filter as a workload optimization, not a security boundary.

This distinction also matters when choosing alternatives. History-aware MCP routing ranks currently eligible tools; it does not authorize them. Likewise, a policy relevance score cannot replace a current entitlement or rule check.

What the research measured

The October 2026 arXiv v1 preprint Signals and Queries describes a per-policy classifier. For each policy, a model generates ten activating messages and ten hard negatives offline. Their embeddings train a small logistic classifier. At a live turn, the system embeds conversation views once, scores each policy, and sends the selected set to the downstream evaluator. A margin trades more included rules for fewer misses.

The evaluation used 254 prefixes from 50 simulated airline conversations and 36 policies; five policies never appeared. Its 9,144 relevance labels came from an LLM judge, not a human-adjudicated panel. At benchmark-selected operating points, the method reached 91–93% macro recall and 32–34% precision across two embedding providers. The paper reports lower first-stage work than a per-policy LLM judge, but these are relevance metrics and measured costs in that setup—not production policy compliance, end-to-end task quality, or a safe omission rate for another inventory.

Build the evaluation around false exclusions

1. Inventory the rules. Mark critical, always-on, and dependency-linked policies before training a filter. Do not let a classifier suppress a rule that must be enforced regardless of conversational wording.
2. Create contrasting examples. Pair activating requests with near-misses from neighboring policies. Review the examples for incorrect labels and missing phrasing, especially in multi-turn or multi-intent conversations.
3. Hold out real workflow cases. Label policy applicability independently of the filter and, where possible, independently of the same judge used to generate or tune its examples. Measure false exclusions per policy and risk tier, not only aggregate recall.
4. Tune the fallback. Include low-confidence policies, revert to unfiltered evaluation, or stop for review when the rule set or conversation is out of distribution. Re-run the held-out set after policy, embedding, or threshold changes.

These are Bonfire operating criteria, not thresholds prescribed by the paper. The paper explicitly notes that independently scored policies do not retrieve a follow-up rule merely because another rule fired. Model that dependency in the orchestration layer, or leave the dependent rule in context.

Release it as a monitored optimization

Run the filter in shadow mode against the existing evaluation path. Log the policy-set version, input case, proposed included and excluded rules, score margin, downstream decision, and adjudicated misses. Compare accepted outcomes and total cost with the unfiltered baseline. Promote only after replaying known edge cases and setting a rollback trigger. The runtime-policy update boundary still governs activation, while a multi-turn tool-policy audit checks whether hidden dependencies escape a single-turn relevance score.

Open Policy Agent's policy-testing documentation provides a practical model for testing rules with explicit inputs and expected decisions. A retrieval benchmark cannot substitute for those enforcement tests. Keep the policy engine or deterministic checks in place even if the prefilter earns its cost savings.

Evidence limits

The study is an unreviewed preprint from the team developing the method. Its conversations come from one simulated airline domain, its relevance reference is an LLM judge without measured human agreement, and its global threshold was selected on the reported benchmark. The reported 91–93% macro recall does not establish recall for rarely activated policies, other domains, languages, or live traffic. Its extension to skills and other context artifacts is a proposal, not an evaluated production result.