How to safely update runtime policies for long-lived AI agents

Treat every policy change as a candidate, not an immediate production rule. Check that the proposed rule is expressible in a trusted event schema, mechanically valid and conflict-free, corrects the observed failure without breaking legitimate decisions, and has trustworthy evidence behind activation. Keep the current policy in force during review; version the activated rule and retain rollback. A formal allow/refuse result is only as sound as the event supplied to the policy engine.

A verified patch is not yet an authorized patch

VEPA's full paper describes an LLM proposing changes to a symbolic policy, deterministic structural and conflict checks, and a separate evidence-aware layer for activation. The authors test it on ASSEBench-Safety, ASSEBench-Security, and ATBench, including mixed and corrupted-feedback streams. These are research results, not a production guarantee.

Suppose an agent wrongly sends a sensitive document to an external recipient. Does a proposed refusal rule match trusted action, data-classification, recipient, and authorization fields—or an incidental string in one trace? Record each field's source and unknown values. A rule can be well formed while the event feeding it is inaccurate. VEPA's reproducible decision applies after an interaction is converted into a fixed typed event; the upstream LLM parser and an LLM fallback for uncovered events are outside that formal guarantee. Its fixed schema and binary decisions also cannot express every temporal or relational requirement.

Test whether a malicious tool response could be mistaken for an authorized instruction. The instruction-hierarchy testing guide addresses that trust boundary; a policy verifier does not replace it.

Replay what the change would break

Test both failures the patch should fix and legitimate decisions the current policy gets right. VEPA tracks both, but verification against observed examples cannot establish behavior for every future event. Keep an auditable release record:

This is a recommended release record, not a VEPA benchmark metric. OPA policy tests can make decisions executable; bundles and status reporting support distribution and observation. None certifies that an agent's typed events or feedback are correct.

Do not promote a rule on the same evidence that suggested it

VEPA's evidence-aware layer delays activation until a verified rule has supporting observations and can remove authority when feedback contradicts it. But its criterion reuses adaptively selected observations; the authors do not give a selection-adjusted or time-uniform guarantee across repeated activation checks. For consequential permission expansion, prefer held-out or prospective examples and explicit reviewer approval. Label feedback by origin and reliability: a user report, benchmark label, tool log, and adversarial message are not interchangeable. If a patch only blocks unsafe action, still measure legitimate-work cost. Sparse or contradictory evidence means the rule stays inactive.

Keep enforcement and rollback observable

Activate the smallest change that addresses the demonstrated case. Pin policy and schema versions for each decision; verify the intended version reached the runtime. Distribution status does not prove every request was classified or enforced correctly. Keep a prior version available and define the contradiction or false-block condition for rollback. The least-privilege delegation guide provides permission context; the agent rollout checklist places the change among other release checks.

Evidence limits

VEPA is an unreviewed arXiv v1 preprint evaluated on named benchmarks and simulated streams, not this reader's production workflow. Determinism depends on a fixed typed event; parser and fallback behavior sit outside it. A fixed schema and binary decision language may miss multi-step requirements. Its evidence-based authority criterion is not a prospective, time-uniform safety guarantee, and no universal sample size or safe threshold follows from the results.