Writing
Mechanism-level pieces on agent reliability, AgentOps, MCP servers, and the harness that keeps AI systems honest in production.
· browser agents · agent safety · AgentOps
Check consequential browser clicks against user intent, target identity, and action policy; confirm high-impact actions and test visible deception separately.
Read →· MCP · agent tools · agent evaluation
Test supplier-authored tool claims against executed capability, then measure whether AI-agent selection delivers the requested result.
Read →· coding agents · agent evaluation · software quality
Test each feature extension against new and earlier requirements, then verify the whole feature on the final workspace rather than scoring only the latest patch.
Read →· coding agents · agent reliability · tool contracts
Check repository-state prerequisites before coding-agent tool calls; block confirmed mismatches while measuring false blocks and task outcomes.
Read →· MCP · agent evaluation · agent reliability
Test MCP-connected agents with seed-recorded error, content, and timing faults; verify exposure and grounded outcomes against pass-through controls.
Read →· agent safety · policy engineering · AgentOps
Use recall-first policy screening to reduce agent evaluation cost while measuring false exclusions and keeping authorization independent.
Read →· coding agents · agent workflows · agent evaluation
Verify each agent edit against one baseline, map shared acceptance conditions, and test the combined artifact before merging parallel changes.
Read →· agent privacy · MCP · agent safety
Carry task-scoped privacy preferences through agent and tool handoffs, enforce data-use decisions at receiving endpoints, and test what intermediaries retain.
Read →· agent safety · agent evaluation · AgentOps
Map shared evidence sources before combining AI monitor approvals; test source-counting against unsafe approvals, false rejections, and escalation needs.
Read →· agent safety · policy engineering · AgentOps
Verify typed policy patches, replay failures and legitimate actions, assess feedback authority, and version activation with rollback.
Read →· agent safety · agent evaluation · AgentOps
Probe multi-turn tool workflows against observable rules, verify alleged violations with full traces, and report unreached states and normal-task utility.
Read →· agent reliability · tool contracts · MCP
Validate calls against the current authorized tool contract; use bounded retry or argument repair for detectable drift, then recheck intent and safety before execution.
Read →· agent safety · agent evaluation · AgentOps
Replay later tasks with and without an agent-authored controller, memory, or skill edit; gate promotion on both task quality and safety.
Read →· agent safety · agent evaluation · AI control
Test an AI-agent monitor against useful-looking artifacts with concealed harmful behavior; verify targeted probes, detection, false alarms, and release thresholds.
Read →· agent safety · agent evaluation · multilingual AI
Replay matched, labelled tool-use steps across user languages and measure guard verdict flips, false positives, and missed unsafe actions before release.
Read →· agent evaluation · knowledge conflict · AgentOps
Pair conflicting and consistent evidence cases; score recognition, bounded resolution, and disclosure separately from answer accuracy and needless hedging.
Read →· multi-agent systems · agent evaluation · AgentOps
Trace source-to-decision handoffs in a failed multi-agent run to distinguish an introduced error from one inherited or repaired.
Read →· agent cost · budget control · AgentOps
Meter confirmed tokens, forecast remaining calls and repeated context, then validate budget interventions against accepted run outcomes.
Read →· agent evaluation · fault injection · AgentOps
Define an authoritative refutation, prove a legal pre-use evidence path exists, then grade each agent trace by when counterevidence actually arrived.
Read →· agent evaluation · agent reliability · AgentOps
Check task coherence, policy challenge coverage, rubric fit, and documentation before treating a conversational-agent benchmark score as release evidence.
Read →· agent safety · agent evaluation · AgentOps
Rerun matched security cases with controlled tool-name variants, then compare attack success and benign utility before relying on a benchmark score.
Read →· agent safety · prompt injection · agent evaluation · AgentOps
Replay representative tool returns, freeze a detector threshold before held-out testing, and measure both attack recall and benign-task blockage before release.
Read →· agent evaluation · enterprise agents · AgentOps
Build read-only cross-system cases with independently computed answer keys, wrong single-record shortcuts, and inspectable evidence paths before using agent scores for release decisions.
Read →· agent safety · runtime protection · AgentOps
Start with audited agent actions, qualify detections against legitimate work, then scope and verify blocking rules while preserving investigation evidence.
Read →· agent safety · agent evaluation · AgentOps
Test complete task-policy conflicts across blocked calls, retries, split actions, and unmonitored paths; score attempted bypass, actual effects, and safe-task utility separately.
Read →· agent evaluation · agent reliability · AgentOps
Screen a candidate under a fixed simulator and evaluator contract, check it against matched production versions, then confirm selected changes in a bounded live test.
Read →· agent safety · agent reliability · AgentOps
Specify approved changes independently, observe the candidate effects inside a controlled boundary, and commit only an exact match while gating retries and downstream work.
Read →· agent evaluation · tool use · AgentOps
Fix a failed-tool observation and its missing completion evidence, then score the agent’s final report for unsupported success and fabricated detail alongside useful recovery.
Read →· agent evaluation · tool use · AgentOps
Record request, serving, model, parsing, and retry outcomes separately before comparing local agents on tool-call fidelity.
Read →· agent safety · authorization · AgentOps
Govern access as a revocable engagement: verify the operator, bind each run to an authorized scope, constrain tools, record evidence, and retain a human stop path.
Read →· agent observability · agent safety · AgentOps
Capture model and tool evidence outside the agent's authority, protect accepted records, and test deletion, rewriting, suppression, and bypass before release.
Read →· MCP · agent safety · agent evaluation · AgentOps
Red-team MCP hijacking in three passes: isolate malicious tool selection, isolate manipulation through tool returns, then test the full chain with separate evidence for each boundary.
Read →· agent safety · agent evaluation · AgentOps
Evaluate private-data exposure as a bounded session property: pre-register governed targets and authorization, measure every declared visible exit, and pair exposure results with normal task success.
Read →· agent safety · agent evaluation · AgentOps
Build a versioned corpus of executable prompt-injection cases, separate discovery from held-out release evidence, and replay protected decisions after every material agent change.
Read →· agent safety · agent evaluation · AgentOps
Warm up representative memory, tools, and shared state, inject one bounded adversarial event, then assess detection, containment, recovery, and cleanup separately before promotion.
Read →· agent evaluation · agent reliability · AgentOps
Set the release rule before the run: version representative cases, define quality and safety criteria, set per-criterion limits and vetoes, then retain the evidence behind the promotion decision.
Read →· agent safety · authorization · AgentOps
For a consequential agent action, a missing policy decision must mean do not perform the action: bound the policy call with a timeout, accept only a validated verdict, and turn a missing, invalid, or failed decision into a durable, recorded deny.
Read →· agent safety · agent workflows · AgentOps
Choose review or autonomous execution for each AI operations workflow based on action boundaries, permissions, recovery, and evidence—not a blanket setting for the entire agent.
Read →· OpenAI · agent evaluation · AgentOps
Treat a moving model alias such as chat-latest as a discovery and evaluation input, not a production pin: gate any change behind a pinned snapshot, a recorded evaluation, a rollout decision, and a rollback path.
Read →· OpenAI · API security · AgentOps
Treat an expiring OpenAI project API key as a planned cutover: inventory consumers, create a scoped replacement before expiry, verify each workload after cutover, and revoke the old key only when the evidence is complete.
Read →· OpenAI · agent workflows · AgentOps
Use the OpenAI Agents API when an OpenAI-managed Codex harness and durable sessions remove mechanics you do not need to operate; keep tool authority, execution-environment choice, application state, and consequential-action policy explicit in your application.
Read →· MCP · gateway · AgentOps
Use the MCP 2026-07-28 Streamable HTTP headers to make a narrow edge classification decision — version, RPC method, and named tool or resource — then let the MCP server reject header/body mismatches, validate the request and token, authorize the operation, and enforce argument- and side-effect-specific controls.
Read →· OpenAI · agent workflows · AgentOps
Use async tool calling only when useful model work can continue while your application owns the slow job — correlate the result by call_id, keep cancellation, compensation, and relevance decisions outside the model, and treat mid-turn steering as queued input, not a rollback of tool work already started.
Read →· MCP · protocol · compatibility
MCP 2026-07-28 removes the initialize handshake for a stateless, per-request protocol — detect a peer's era with the documented probes, keep authorization failures separate from version negotiation, and hold traffic until the still-unreleased revision is verified.
Read →· OpenAI · agent reliability · AgentOps
Branch retry control on the documented failure meaning, not the HTTP status: reduce ramp after a 429 slow_down, use exponential backoff for a temporary 503 server_is_overloaded, and honor Retry-After when either is present.
Read →· agent evaluation · agent safety · AgentOps
Treat an agent evaluation as an untrusted security zone: use a separate identity, default-deny egress and shared-service access, broker only required dependencies, and test escape paths before the run.
Read →· agent memory · agent safety · AgentOps
Keep provenance and confirmation attached through memory consolidation, bind that evidence to action-relevant tool arguments, and require authority appropriate to the action risk.
Read →· agent safety · agent workflows · MCP · AgentOps
Treat each agent handoff as an explicit, bounded delegation: narrow the resource and scope, set expiry and budget, then enforce the envelope outside the model before a consequential tool action.
Read →· agent safety · MCP · cryptographic signing · AgentOps
Keep a private signing key inside hardware, then require identity, scope, semantic, and taint checks before an AI agent's request can reach the signature boundary.
Read →· agent evaluation · agent reliability · AgentOps
Evaluate a state-changing agent by its durable postcondition, allowed execution path, and prohibited side effects—then repeat controlled cases before treating the workflow as dependable.
Read →· voice agents · agent reliability · AgentOps
Keep the live media loop separate from slow background work, retain authoritative session and job state outside it, and make pending, cancellation, recovery, and relevance decisions explicit.
Read →· coding agents · agent workflows · AgentOps
Treat a specification from one coding agent as an input the receiving agent must accept against representative work and required checks; retain the handoff only when its implementation meets the agreed acceptance criteria.
Read →· OpenAI · agent observability · cost controls · AgentOps
Baseline cache reads and writes by model and service tier, then change one request-shape or routing variable at a time; retain a change only when cache evidence and accepted task outcome support it.
Read →· agent memory · agent reliability · AgentOps
When validated evidence supersedes a memory under a stable key, remove the old record from ordinary retrieval, preserve its revocation trail for audit, and expose only active evidence by default.
Read →· agent observability · agent safety · AgentOps
Treat agent telemetry as a governed data-export path: keep content capture deliberate, separate restricted traces from broad health signals, and test where representative sensitive data actually lands before release.
Read →· agent evaluation · agent reliability · AgentOps
Compare a compressed run with a matched full-context baseline, then measure accepted completion, retrieval and execution work, latency, and cost separately before widening the policy.
Read →· agent evaluation · agent reliability · AgentOps
Keep the release decision outside the optimization loop: reserve acceptance evidence, meter the search budget, retain the exact candidate and its evaluation record, then run an independent release check.
Read →· agent memory · agent reliability · AgentOps
Manage reusable agent experience as a feedback-controlled asset: admit it selectively, preserve provenance and later outcome evidence, and retire records whose reuse is associated with poor downstream results.
Read →· OpenAI · cost controls · AgentOps
Assign one documented API key to one workload, then use OpenAI API-key filtering and grouping to inspect provider-recorded usage and cost without mistaking that ledger for full workflow economics.
Read →· agent reliability · observability · AgentOps
Find the earliest failure that matters by reviewing an evidence-backed error lifecycle—not by selecting the first bad-looking step or the final symptom.
Read →· coding agents · agent reliability · AgentOps
Treat a user edit as a plan-invalidating state change: pause writes, preserve and inspect the diff, identify affected assumptions and checks, then re-plan or escalate before continuing.
Read →· MCP · agent reliability · AgentOps
Use interaction history in MCP tool routing only after resolving the current authorized tool set: record the history and candidates behind a choice, retain a stateless fallback, and test whether added state improves accepted outcomes.
Read →· agent safety · agent evaluation · AgentOps
Test instruction hierarchy at the action boundary: declare the protected constraint, inject user and tool-output conflicts, and assert an allowed action, refusal, or escalation before a side effect occurs.
Read →· OpenAI · agent reliability · AgentOps
Treat OpenAI Fast mode as an observable latency tier: request it deliberately, ramp gradually, inspect the returned service tier, and pause expansion when the returned path does not match the request.
Read →· agent workflows · AgentOps · orchestration
Use multiple agents only when a bounded relay preserves the facts the next decision needs; otherwise keep shared context or redesign and test the relay against a single-agent baseline.
Read →· OpenAI · cost controls · AgentOps
Treat an OpenAI hard spend limit as an intentional availability boundary: classify the quota condition, preserve the workflow checkpoint, and explicitly decide whether to restore service or wait.
Read →· agent reliability · AgentOps · agent evaluation
Stop a verify–repair loop when evidence no longer supports another revision: preserve the champion, enforce a budget, and escalate when the verifier or calibration cannot support a reliable choice.
Read →· agent evaluation · AgentOps · agent reliability
Evaluate an AI agent reviewer as a chain: correct detection, critique uptake, accepted repair, and cost. Reviewer precision alone does not prove that the final answer improved.
Read →· MCP · security · agent reliability
Keep a private MCP server non-public while giving an agent a controlled connection path—then retain authorization, tool policy, auditability, and recovery at the server boundary.
Read →· agent workflows · AgentOps · orchestration
Agent loops remain useful for bounded work. When a system needs parallel branches, approval gates, checkpoints, and scoped recovery, make the control flow explicit with an agent workflow graph.
Read →· OpenAI · agent security · identity
Use OpenAI Workload Identity Federation to exchange an external workload identity for a short-lived access token, with a narrow service-account mapping and explicit permissions.
Read →· observability · agent reliability · AgentOps
An agent observability dashboard should show run state, tool behavior, retry pressure, and spend together so operators can explain the workflow before they intervene.
Read →· agent reliability · AI readiness · AgentOps
An AI agent rollout checklist should prove scope, tool safety, recovery, review, telemetry, and cost controls before the workflow runs without a human watching every step.
Read →· AgentOps · incident response · agent reliability
AgentOps incident response starts by freezing side effects, preserving state, classifying the failure, and choosing a recovery path before the agent resumes.
Read →· agent reliability · observability · failure modes
AI agents fail in production for boring harness reasons: vague tool contracts, lost state, bad retries, and missing observability.
Read →· MCP · tools · production
A production MCP server survives when tools, resources, authorization, and logging are explicit and the server owns the audit trail.
Read →· AgentOps · agent reliability · SRE
Treat production AI agents like SRE systems: checkpoints, idempotent writes, retry budgets, observability, and cost guards around the model.
Read →· agent reliability · governance · approvals
Human approval gates work when the harness freezes the exact action surface, preserves the reviewer decision, and invalidates stale approvals before resume.
Read →· agent reliability · recovery · checkpoints
An agent workflow recovery plan reconciles checkpoints, side effects, and approval state before a resumed run is allowed to act again.
Read →· AgentOps · agent reliability · SLOs
Agent reliability SLOs should measure finished work, recovery behavior, escalation, and cost per accepted output rather than model latency alone.
Read →· cost controls · agent reliability · runbooks
A cost runbook classifies the loop, contains the damage, and measures accepted output so routing and retry controls get tuned for the right failure mode.
Read →· agent reliability · governance · controls
A kill switch should pause the workflow, preserve the last safe checkpoint, and hand the decision back to a human before the agent keeps acting.
Read →· agent reliability · tool contracts · idempotency
A production agent tool contract separates intent, validation, execution, and audit so retries stay legible and side effects stay bounded.
Read →· agent reliability · AI readiness · scoping
An AI agent scope audit chooses the first workflow by recoverability: clear ownership, bounded tools, observable progress, and reviewable side effects.
Read →· AgentOps · agent reliability · services
AgentOps hardening services inspect workflow contracts, tool safety, recovery behavior, telemetry, and review boundaries before a production agent runs unattended.
Read →· cost controls · agent reliability · routing
Multi-model routing controls cost by making the harness choose the cheapest reliable lane for each agent step, then measuring accepted output instead of raw call spend.
Read →· agent reliability · idempotency · MCP
Make tool calls idempotent by separating intent, side effect, and acknowledgement — so a retry never double-charges, double-sends, or double-writes.
Read →· AgentOps · agent reliability · production AI
AgentOps is the discipline of making AI agents reliable in production — checkpoints, idempotent tool calls, retry budgets, cost guards, and observability. The mechanism, the metrics, and a hardening checklist.
Read →