AI agent threat modeling: A practical guide

What is AI agent threat modeling?

AI agent threat modeling identifies how an attacker could turn an agent's inputs, permissions, memory, or tools into an unwanted action. A threat model records concrete failure scenarios, the controls intended to stop them, and tests that verify each control. Start with what the agent can change or disclose, then trace backward to the sources that could influence that action.

An agent that summarizes public documentation has a different threat model from an agent that reads customer records and sends email. The model may be identical. The difference is the authority and data available to the surrounding application.

The OWASP agentic threats and mitigations resource is a useful reference for threat identification. The worked following example is an illustrative design exercise, not a report of an Arcjet customer incident.

Inventory actions, identities, and data

Consider a support agent that retrieves tickets, reads billing records, drafts replies, and issues refunds. Before selecting a security product, record the following details:

List background jobs and retries as well as interactive requests. A refund that is queued during a conversation may execute after the user's access has changed. The worker still needs to authorize the operation when it runs.

Mark where trust changes

Trace the workflow as a sequence: authenticated request, retrieval, model context, proposed tool arguments, authorization, execution, and stored results. At each transition, identify who supplied the value and who is allowed to make it authoritative.

A ticket can provide facts for a reply. It cannot establish a new recipient or approve a refund. A model can propose an invoice ID. It cannot establish ownership of that invoice. A tool result can describe an error. It cannot grant a broader credential for the retry.

Keep identity, tenant membership, approvals, and policy versions outside model-authored fields. Where two agents collaborate, preserve that separation across the handoff. A supervisor's summary is still content to validate; it is not a new source of authorization. The on-behalf-of identity guide explains how to preserve user and agent identity across services.

Write attack scenarios with observable outcomes

Use this sentence template: "An actor who controls this input can influence this decision, causing this effect unless this control intervenes." Avoid ending the scenario at "the model follows an instruction." That observation matters only when connected to disclosure, integrity, availability, or an unauthorized operation.

For the support agent, useful scenarios include:

• A customer edits a ticket to request that billing history be sent to a new address. The reply tool must reject recipients outside the verified case.
• A retrieved document tells the agent to use an invoice from another tenant. The billing service must reject the object reference even if the tool arguments are valid JSON.
• Two parallel refund calls each pass a balance check. The payment path must enforce the remaining amount atomically and deduplicate retries.
• An old memory entry claims that a supervisor approved all future refunds. The application must require a current approval tied to the actual operation.
• A denied tool call is retried through a second tool with equivalent access. Both paths must apply the same business authorization.

These scenarios expose different gaps. A prompt-injection detector may flag the first input, but it cannot substitute for recipient binding, tenant authorization, concurrency control, or approval integrity.

Prioritize by reachable consequences

For each scenario, record the attacker access required, affected data or action, control in place, and remaining uncertainty. Prioritize a customer-editable ticket that can reach a payment tool over a hypothetical model exploit with no path to sensitive resources.

Review combinations as well as individual tools. Read access to private data and permission to send an external message create an exfiltration path even when neither capability is obviously dangerous in isolation. See the lethal trifecta and data exfiltration guide.

Assign an owner to each control and give unresolved assumptions an explicit status. "The SDK probably checks this" is an open question. "The integration test observes zero payment calls after denial" is evidence. Refresh the model whenever tools, credentials, retrieval sources, memory scope, or execution environments change.

Turn the threat model into a release gate

Build a fixture for each scenario using synthetic records and instrumented tool implementations. Observe the actual side effect, not just the assistant's final message. A response that says "I cannot do that" is insufficient if a payment already executed earlier in the run.

Test the authorization function directly and then exercise it through the agent. The first test isolates the control; the second proves the agent framework reaches it. Include denial, timeout, retry, and permission-revocation cases. Use the AI agent security testing guide to organize the resulting suite.

Arcjet Guards provide a place to evaluate policy at integrated tool boundaries. Your application still supplies trusted context, checks resource ownership, and handles the decision before executing. Start with one consequential tool, apply least privilege, and verify every alternate path to the same action.

For a complete rollout sequence, continue with securing AI agents in production.