AI security checklist for production LLM apps

The AI security checklist

Most AI security checklists are governance artifacts: a list of policies to write and committees to form. This one is a list of things you can verify by reading code or a decision log. Each item is answerable yes or no, and a no tells you what to build.

The grouping follows the four functions of the NIST AI Risk Management Framework, so the list maps onto the framework your risk or compliance colleagues are working from. The items themselves are runtime controls: checks that return a decision before an action happens. For the wider category and how the layers divide, see what is AI security.

GOVERN

• Every AI feature has a named owner who can change its security controls.
• Security rules for AI features live in version control, reviewed in the same pull request as the feature.
• Each control has a documented behavior when its dependency is slow or unreachable, decided per action rather than globally.
• The agent's authority is scoped per audience, and its identity is separate from the user's.
• A tool added to an agent cannot reach production without passing through the same policy as the tools already there.

MAP

• You have a list of the consequential actions each AI feature can take, derived from recorded activity where you have it, rather than a list of routes.
• For each action, you know which data it reads, whose authority it uses, and whether it can be undone.
• You know every path by which untrusted text reaches model context, including retrieved documents, tool results, and subagent output.
• You know which actions can communicate externally, and which of those also see private data.
• Your dependency inventory covers third-party model providers, MCP servers, and agent frameworks, each with an owner.

MEASURE

• Every probabilistic control ran in dry run against your own traffic before it started blocking.
• You have a false-positive rate for each detection control, measured on your workload rather than a vendor benchmark.
• A test proves that each denied action's side effect does not execute.
• You can tell, for any request, which controls evaluated it and what each one returned.
• You know the added latency of each control on the paths where it runs.

MANAGE

• The check for each consequential action runs immediately before the side effect, in the same function.
• Denials are distinguishable to the caller, so over budget and content rejected are different responses, without handing back the rule that fired.
• Every decision is recorded with enough context to explain it to a customer, including what the application did next.
• Token and spend budgets are keyed on an identity the caller cannot change.
• You have exercised the path where a control's dependency is unavailable, and the observed behavior matches the documented one.

The rest of this article explains why each group holds together, and maps the OWASP LLM risks onto the same controls.

What NIST AI RMF and OWASP each give you

These two documents are complementary, and neither one tells you where the code goes.

The NIST AI Risk Management Framework (AI RMF 1.0) is a voluntary framework for organizing AI risk work. Its core has four functions: GOVERN, MAP, MEASURE, and MANAGE, each with categories and subcategories. GOVERN is cross-cutting rather than sequential: it applies across the other three functions rather than preceding them. NIST also publishes a Generative AI Profile that enumerates risks specific to generative systems, and a companion Playbook of suggested actions. The framework tells you what to govern, measure, and manage. It's deliberately technology-neutral, so it doesn't name a control.

The OWASP Top 10 for LLM Applications goes the other way. It enumerates what actually goes wrong in production: prompt injection, sensitive information disclosure, excessive agency, unbounded consumption, and the rest. It's specific about failure modes and lighter on the enforcement point.

So the gap is the same in both cases. A framework says measure your system's security; a risk list says prompt injection is the top risk. Neither says the check belongs inside the tool handler, before the write, keyed on an identity the model can't set. That last sentence is the part that changes your code, and it's what the checklist is for.

One clarification worth having ready when this comes up with a customer or an auditor: the AI RMF is not a certification. There's no such thing as being NIST AI RMF certified. ISO/IEC 42001 is the certifiable AI management-system standard. For what an auditor asks for in practice, see compliance evidence for AI agents.

GOVERN: ownership and failure behavior

GOVERN is where most AI security programs are strongest on paper and weakest in the code path. The two items that matter most to an engineer are ownership and failure behavior.

Ownership means one person can change a control and knows what breaks when they do. That's why the rules belong in version control next to the feature: a rule in a separate console drifts from the application it defends, and teams usually discover the drift during an incident. For the trade-offs when policy has to be centrally managed, see application-native vs remote security policies.

Failure behavior is the item teams skip, and it's the one that decides whether a control survives its first outage. Fail-open keeps the application available with the control off. Fail-closed keeps the control on and the feature unavailable. Neither is right everywhere: failing open on a payment tool during an attack is bad, and so is failing closed on a search feature because a detection API is slow.

Decide it per action, write the decision down, then run the path. An untested failure mode is a guess.

MAP: inventory the consequential actions

The MAP function asks you to establish context and categorize what the system can do. For an LLM application, the useful unit is the consequential action, not the route or the endpoint.

Route inventories miss the paths that matter in agentic systems. A tool handler, a queue consumer, a scheduled job, and a subagent call all take actions with no HTTP request for anything at the perimeter to inspect. So list operations: which of them spend money, move data, send messages to people, change permissions, or can't be undone.

Then do the same for inputs. Every path by which text reaches model context is an injection path, and the indirect ones are the ones teams miss: a retrieved document, an MCP tool result, a webhook payload, a filename. For more information, see indirect prompt injection in agentic workflows.

The output of MAP is a short table you can hand to a new engineer: action, data it reaches, authority it uses, reversible or not. Everything in MEASURE and MANAGE keys off that table.

You don't have to write that table from memory. Where agent activity is already recorded, the inventory comes from what your agents do rather than from what you believe they do. Arcjet's observability features can ingest and render that activity without a change to application code, so the MAP work can start before any rule exists.

MEASURE: prove the control works on your traffic

Detection controls are probabilistic. A prompt-injection classifier, a sensitive-information detector, and a bot classifier all return a judgment that can be wrong in both directions, so the number you need is a false-positive rate on your own traffic.

A vendor benchmark isn't that number. Your traffic contains your users' phrasing, your document formats, and your domain vocabulary, and a security-adjacent product with technical users generates text that looks adversarial while being entirely legitimate.

The mechanism is dry run: deploy the control so it evaluates and records without enforcing, leave it long enough to cover a representative period, then read the decisions. A control that would have blocked 40 real customers in a week is a control you tune before it blocks anyone.

Two more MEASURE items are easy to skip and cheap to add:

• A test that proves the denied action didn't run. Assert on the side effect, not on the decision object. A test that checks the decision was DENY passes even if the write happened anyway. For patterns, see functional testing for security rules.
• Latency per control, on the paths where it runs. A check in a tool loop runs once per iteration, so a cost that's invisible on a page load is not invisible there.

MANAGE: enforce at the boundary and record the decision

MANAGE is where the checklist becomes code. The property that makes a control preventive rather than advisory is that the decision arrives before the effect, in the same function as the effect.

The following example does that with Arcjet, which installs as a library and evaluates its rules in your own process rather than at a network hop. launchArcjet creates the client, each rule is configured once when the module loads, and arcjet.guard() runs them against one call and returns a single allow-or-deny decision. The protected action is an email send, screened for prompt injection in the body and bounded by a per-user token budget.

1
import {
2
  detectPromptInjection,
3
  launchArcjet,
4
  tokenBucket,
5
} from "@arcjet/guard";
6

7
const arcjet = launchArcjet({ key: process.env.ARCJET_KEY! });
8

9
const injection = detectPromptInjection();
10
const budget = tokenBucket({
11
  refillRate: 2_000,
12
  intervalSeconds: 3_600,
13
  maxTokens: 5_000,
14
});
15

16
// What the caller is told, keyed by the reason the decision carries. The
17
// caller of a tool is often the model, so this mapping is a deliberate
18
// choice rather than a pass-through of the security decision.
19
const callerError: Record<string, string> = {
20
  RATE_LIMIT: "over_budget",
21
  PROMPT_INJECTION: "content_rejected",
22
};
23

24
export async function sendCustomerEmail(
25
  args: { to: string; body: string },
26
  session: { userId: string },
27
) {
28
  const decision = await arcjet.guard({
29
    label: "tools.send-customer-email",
30
    actor: session.userId,
31
    rules: [
32
      injection(args.body),
33
      budget({ key: session.userId, requested: 1 }),
34
    ],
35
  });
36

37
  // This action is one of the fail-closed ones, so an incomplete check
38
  // stops the send rather than letting it through.
39
  if (decision.conclusion === "DENY" || decision.hasFailedOpen()) {
40
    logger.warn("guard denied", {
41
      label: "tools.send-customer-email",
42
      reason: decision.reason,
43
    });
44
    return { error: callerError[decision.reason] ?? "blocked" };
45
  }
46

47
  // The send runs only past the deny. Nothing above it has side effects.
48
  return mailer.send(args);
49
}

Three properties of that shape are what the checklist items are checking for:

1. The check is inside the function that does the work. A gate in middleware, a framework permission callback, or a supervisory agent running alongside can all be skipped by a path that doesn't route through them. For why callbacks in particular are weaker than they look, see canUseTool is not a policy gate.
2. The denial is specific, and mapped. The caller, whether a person, a retrying client, or another agent, has to be able to tell "over budget" from "content rejected", because a generic failure is a control someone routes around. It doesn't need the rule that fired: the caller of a tool is often the model itself, and naming the control that stopped it hands an attacker-steered model something to iterate against. So map the reason to a caller-facing code and keep decision.reason in the log.
3. The budget key is server-side. session.userId comes from your session, not from a model-supplied argument or a request header. A limit keyed on something the caller controls is not a limit.

The recording half of MANAGE is what turns enforcement into evidence. A decision log that captures the rule, the identity, the conclusion, the reason, and what the application did next answers both the incident question and the audit question. For more information about the evidence side, see compliance evidence for AI agents.

OWASP LLM risks mapped to runtime controls

The following table maps each OWASP entry to the control that addresses it, the boundary where that control runs, and the RMF function the checklist item sits under.

Most rows need a control that runs while the application is running. For per-entry implementation detail, see how to implement the OWASP Top 10 for LLM Applications.

How to work through the checklist

Working the list top to bottom fails, because the GOVERN items depend on knowing what you have and the MEASURE items depend on something being deployed. Use this order instead:

1. Write the action table. One row per consequential action, with the data it reaches, the authority it uses, and whether it can be undone. Ingest agent activity first if you can, so the rows come from observed behavior rather than recollection. This is the MAP work, and everything else reads from it.
2. Pick the worst row. The action where a failure costs the most, not the one that's easiest to instrument.
3. Put a check in that function. Immediately before the side effect, in the same code path, with the identity and arguments in scope.
4. Run it in dry run and read the decisions. Long enough to cover a representative period of your traffic, before it blocks anything.
5. Decide and test the failure behavior for that action. Fail-open or fail-closed, written down, then exercised.
6. Enforce, with a specific denial and a recorded decision. The caller can tell why, and you can explain it later.
7. Widen to the next row. Repeat down the action table, reusing the same wrapper so the policy is one implementation rather than one per tool.

The GOVERN items then follow from work you've already done: the owner is whoever wrote the wrapper, and the reviewed-in-code property comes free from the rules living next to the feature.

Where Arcjet fits

Arcjet covers the MANAGE items and part of MEASURE, and it can do a useful part of MAP before you install anything.

MAP, with no code change. Point an OpenTelemetry exporter at Arcjet's OTLP endpoint, or connect Anthropic's Compliance API with one access key, and agent activity is recorded and rendered: which sessions ran, when, and by whom, plus the prompts and tool calls themselves when the connection uses a Compliance Access Key rather than an Admin API key, which reaches the activity feed only. Both paths land in the same capture events and the same console views as everything else, so the action table can be built from observed behavior. This is the part of the checklist you can do on day one, before a single rule exists.

MANAGE and MEASURE, in code. Guards put a decision inside a tool handler, a queue consumer, or a background job, which is where the consequential actions in your MAP table actually run. Prompt-injection detection screens inbound text and content re-entering context. Sensitive-information detection classifies the body in your own process, so the data you're protecting doesn't leave to be scanned. Token budgets are token buckets keyed on an identity you pass in. DRY_RUN is a rule mode rather than a fourth conclusion, so a dry-run denial is recorded without changing the allow, which is what makes the MEASURE items measurable. Capture events record what the application did after each decision.

What stays yours is GOVERN, and the judgment inside MAP. Ingested activity shows you what your agents do; it can't tell you which of those actions are the consequential ones, who owns each feature, or whether a payment path fails open or closed. Those are the items the rest of the checklist reads from.

Summary

An AI security checklist is only useful if each item is verifiable. Group the items by the NIST AI RMF functions so the list lines up with the framework your risk colleagues use, but write each item as something you can check in code or in a decision log: the owner of a control, the failure behavior you tested, the dry-run measurement on your own traffic, the check that sits inside the function that does the work. The OWASP Top 10 for LLM Applications tells you what goes wrong and the AI RMF tells you what to govern. What neither says, and what the checklist adds, is that the decision has to arrive before the effect, in the code path of the action, keyed on an identity the caller can't set.

Learn more: Arcjet Guards · AI runtime protection