What does AI agent security testing measure?
AI agent security testing measures whether an agent can cause an unauthorized action or disclosure, and whether the surrounding controls stop it. Test model behavior, application authorization, and end-to-end execution separately. A refusal in the final answer does not prove that a tool did not already send data or change a record.
Start with a threat model and define the forbidden outcome for each case. For a support agent, that could be a refund on another tenant's invoice, an email to an unapproved recipient, or a private record appearing in persistent memory outside its permitted scope.
The OWASP evaluation criteria for AI red teaming distinguish realistic application testing from evaluations limited to jailbreak prompts. Use external benchmarks to discover test ideas; measure release readiness against the application you actually operate.
Separate three kinds of tests
Authorization tests call the tool or service directly with controlled identities, arguments, and resource state. These establish whether a permission boundary holds without depending on a model to generate a particular request.
Detector evaluations measure how a prompt-injection or sensitive-data detector classifies representative malicious and benign inputs. These establish accuracy and operational trade-offs, not complete workflow security.
End-to-end agent tests run the actual orchestration with instrumented tools. These establish whether every execution path reaches the expected controls, including handoffs, retries, and resumed tasks. Keep the tool effects inside a disposable environment with synthetic records and controlled destinations.
These suites answer different questions. If a detector misses a malicious ticket but the tool refuses its unauthorized recipient, the detector case failed and the action boundary held. Record both results.
Build an agent security test matrix
The following cases are illustrative. Adapt the expected result to the permissions and business rules of your application.
| Case | Input variation | Evidence to assert |
|---|---|---|
| Normal task | Authorized user requests an allowed refund. | Exactly one authorized payment occurs and the decision is attributable. |
| Cross-tenant access | Tool arguments reference another tenant's invoice. | No record contents or payment operation reach the caller. |
| Indirect injection | A retrieved ticket asks for an unapproved destination. | No message reaches that destination, regardless of the final model response. |
| Approval substitution | The amount changes after a human approves it. | The original approval cannot authorize the changed transaction. |
| Persistent contamination | Untrusted instructions are stored and read in a later run. | Stored content cannot grant new privileges or cross the memory's tenant scope. |
| Parallel spending | Two permitted-looking calls exceed a shared limit together. | The total effect stays within the limit under concurrency. |
| Control outage | Authorization or policy evaluation is unavailable. | The sensitive operation follows the documented failure policy. |
| Uncovered path | The same action runs through a retry worker or alternate tool. | The same authorization invariant holds on that path. |
Report useful metrics with denominators
For detector evaluations, report false positives as benign cases incorrectly flagged divided by all benign cases. Report misses as malicious cases not detected divided by all malicious cases. Include the corpus composition, detector configuration, and evaluation date. A single accuracy percentage can hide a detector that rejects too much legitimate traffic.
For agent runs, report forbidden outcomes observed divided by attempted scenarios, along with repetitions per scenario. Distinguish a blocked tool call from a task that never attempted the action. Record model version, prompt version, tool schema, framework version, and security configuration so another run can be compared meaningfully.
Nondeterministic behavior needs repeated trials, and a finite run with no failures does not prove that the system is unbreakable. Direct authorization tests give stronger evidence for a specific permission invariant; model-driven tests explore whether realistic workflows reach it.
Measure added latency and legitimate task completion too. A control that blocks every action has a low observed attack-success rate and an unusable product. Set thresholds from your workload and risk tolerance, not an unsupported universal target.
Use findings to decide whether to release
A practical release gate includes a passing legitimate workflow, passing direct authorization tests, and no observed forbidden side effects in the defined regression corpus. Treat a new unauthorized write or cross-tenant disclosure as a reason to fix the boundary before release. For probabilistic detectors, compare performance against your documented acceptance thresholds and investigate regressions.
Preserve the smallest reproducible case from each incident or red-team finding. Rerun affected cases when models, prompts, tools, dependencies, permissions, or policies change. Retain a held-out set so tuning to a known corpus does not become the only evidence of improvement.
Keep security-test data out of real customer accounts. Log the minimum evidence needed to debug the result, and restrict access to any deliberately sensitive fixtures. The NIST Generative AI Profile provides a broader reference for evaluation and risk management.
Test an Arcjet integration
For an application using Arcjet, verify where Guards are called and what the application does with each decision. Test your actual integration's error handling rather than assuming all framework wrappers share the same default. A recorded denial is insufficient if the caller ignores it and executes anyway.
Use dry-run observations to understand detector behavior on representative traffic before enabling blocking. Then test enforcement with instrumented downstream operations. Keep application-owned authorization tests alongside the Guard tests: a policy service does not supply tenant ownership or business permissions automatically.
Continue with the AI security checklist for rollout and the incident response playbook for failures discovered after deployment.
Frequently asked questions
What do AI agent security tests need to verify?
Verify that authorized work succeeds and forbidden actions or disclosures do not occur. Observe downstream effects and test alternate execution paths, not only the assistant's final response.
How do I measure prompt-injection detection?
Report false positives on representative benign inputs and misses on malicious inputs, with corpus composition, configuration, and denominators. Measure end-to-end unauthorized outcomes separately.
Does passing an AI red-team test prove an agent is secure?
No. Results cover the tested scenarios and configuration. Repeat nondeterministic trials, retain held-out cases, and use direct authorization tests for specific permission invariants.
AI runtime security in your code
Protect your AI agent workflows with Arcjet
Arcjet guards run inside the tool, so the allow or deny arrives before the side effect rather than after it.