AI agent incident response: A practical playbook

How do you respond to an AI agent security incident?

AI agent incident response starts by containing the agent's ability to take further actions. Preserve evidence, determine which downstream effects occurred, and repair the failed control before restoring access. Include queued work, credentials, persistent memory, and connected tools in containment. Stopping the chat interface alone may leave the agent's other execution paths active.

An AI agent incident can begin with a suspicious prompt, but the operational question is what the system read, disclosed, changed, or scheduled. A prompt-injection alert may indicate an attempt that was blocked. A normal-looking tool call may reveal a successful unauthorized action. Investigate both the decision and the effect.

The NIST Generative AI Profile includes incident response and recovery planning for generative AI systems and their downstream participants. The following playbook applies those responsibilities to an illustrative support-agent incident.

Contain execution and delegated access

Suppose a support agent sends billing details to a recipient found in an untrusted ticket. Begin by disabling the affected send capability or pausing the workflow. Identify active runs, queued messages, scheduled continuations, retry workers, and subagents that share the capability.

Revoke or narrow affected credentials where necessary. Check whether downstream services honor revocation immediately or only after an issued token expires. Remove access at the resource or network boundary if stopping the orchestrator cannot reliably contain it.

Choose the smallest scope that contains the incident: one run, agent version, tenant integration, tool, or service account. If isolation cannot be established, widen containment. Record the operator, time, and scope of every containment action so investigators can distinguish attacker activity from the response.

Don't depend on the compromised agent to stop itself. Use controls owned by the application, identity provider, queue, or downstream service.

Preserve a timeline without spreading sensitive data

Capture identifiers and versions before replacing the configuration. Preserve raw content only where it is necessary for investigation, with restricted access and an appropriate retention policy. Don't copy leaked data into an incident ticket unless the investigation requires it.

Treat captured prompts, documents, and tool results as untrusted evidence. If an analyst uses an AI assistant to summarize them, keep them separate from operational instructions and deny that assistant unnecessary tools or credentials. See compliance evidence for AI agents for the distinction between decisions and execution records.

Determine the effects and persistence

Trace backward from the unwanted operation to the first controllable input. Then trace forward through every use of the affected credential, resource, or stored memory. Check sibling runs if they consumed the same document or shared the same permission.

For the support-agent example, verify the actual recipient, which fields were sent, and whether the tool sent additional messages after the first one. Inspect pending sends and retries. Determine whether the malicious ticket text entered persistent memory or a retrieval index that other agents will read.

Separate confirmed effects from hypotheses. "The agent requested an export" and "the export reached an external destination" are different findings. Record missing telemetry as an uncertainty rather than filling the gap with the model's account of what it did.

Assess notification and escalation through your security, privacy, and customer-response process using the confirmed data and impact. The agent's involvement does not replace that process.

Repair the boundary that allowed the effect

If a ticket changed the email destination, bind recipients to verified case data. If a tool read another tenant's record, repair object-level authorization. If a retry repeated a payment, fix idempotency and reconciliation. Changing a system prompt may help model behavior, but it does not repair these application invariants.

Quarantine contaminated memory, documents, or tool configurations so they cannot reintroduce the incident. Rotate exposed credentials and check where they were used. Where an external effect cannot be undone, record that limit and route remediation through the owner of the affected system.

Convert the incident into a regression test that checks the real side effect. Include the alternate path that bypassed the original control and at least one normal workflow to guard against an overly broad fix. Use the security testing guide to record the evidence.

Restore service with explicit exit criteria

Before restoring service, test the repaired control, review pending work, and verify that credentials and persistent state are safe to reuse. Then resume the smallest affected scope. Start with restricted permissions and monitor the repaired path. Identify who can pause it again if the same pattern returns.

Arcjet decision records and application capture events can contribute to the timeline for integrations that emit them. A correlation identifier helps connect events; it is not a kill switch or proof that every action was observed. Your orchestrator and downstream services still own cancellation, revocation, and rollback.

Close the incident by updating the threat model, least-privilege matrix, and operational runbook. Keep the test and the control owner attached to the finding so the next tool or framework upgrade does not quietly reopen it.