Prompt injection protection for LangChain, LlamaIndex, and Vercel AI SDK apps

Prompt injection protection for LangChain, LlamaIndex, and Vercel AI SDK apps

The framework changes how you wrap a tool. It doesn't change where the decision has to sit.

LangChain, LlamaIndex, and the Vercel AI SDK all pass model-generated arguments into an execute or invoke function, and they all feed tool results back into the next prompt. A jailbreak in the chat box is one hop. A planted instruction in a fetched page, a loaded document, or a retriever chunk is another. Protection that sits only on the HTTP route misses the second hop.

Not every framework ships a first-party security adapter. The Vercel AI SDK and LangChain have documented wrappers that insert a checkpoint between generated arguments and the tool. LlamaIndex doesn't. A LlamaIndex app still puts the same two checks in the same two places: on the route that accepts the user message, and inside the retriever or tool that returns text to the model.

A production chat route needs more than a system-prompt warning. Hostile instructions try to override the system prompt, and legitimate messages can still carry data that you don't want in model context. The route is also a public HTTP endpoint, so combine a web-attack filter, prompt-injection detection, and a sensitive-data check before you call the provider.

The following example puts all three checks on a Vercel AI SDK route:

1
import { openai } from "@ai-sdk/openai";
2
import arcjet, {
3
  detectPromptInjection,
4
  sensitiveInfo,
5
  shield,
6
} from "@arcjet/next";
7
import type { UIMessage } from "ai";
8
import { convertToModelMessages, isTextUIPart, streamText } from "ai";
9

10
const aj = arcjet({
11
  key: process.env.ARCJET_KEY!,
12
  rules: [
13
    shield({ mode: "LIVE" }),
14
    detectPromptInjection({ mode: "LIVE" }),
15
    sensitiveInfo({ mode: "LIVE", deny: ["CREDIT_CARD_NUMBER", "EMAIL"] }),
16
  ],
17
});
18

19
export async function POST(req: Request) {
20
  const { messages }: { messages: UIMessage[] } = await req.json();
21
  const lastMessage = (messages.at(-1)?.parts ?? [])
22
    .filter(isTextUIPart)
23
    .map((p) => p.text)
24
    .join(" ");
25

26
  const decision = await aj.protect(req, {
27
    detectPromptInjectionMessage: lastMessage,
28
    sensitiveInfoValue: lastMessage,
29
  });
30

31
  if (decision.isDenied()) {
32
    return new Response("Please rephrase your message", { status: 403 });
33
  }
34

35
  const result = await streamText({
36
    model: openai("gpt-4o"),
37
    messages: await convertToModelMessages(messages),
38
  });
39
  return result.toUIMessageStreamResponse();
40
}

Keep denied responses generic. Don't explain what the detector flagged. Measure false positives on live traffic before you start blocking. LangChain in Python is the same shape: run the check on the FastAPI or Flask route, then ainvoke the chain only after an allow. LlamaIndex is the same shape without a wrapper package.

For more information about the three runtime failure modes (injection, exfiltration, and unsafe actions), see runtime security for LLM applications.

How to defend against indirect prompt injection in agentic workflows

Treat every tool result, retrieved chunk, and fetched page as untrusted, and run prompt-injection detection on that text before it goes back to the model. Direct jailbreaks in the user box are only the first hop.

In a normal API, JSON returned from a service is data. In an agent workflow, that same JSON becomes context. A summary, reason, or free-text field can function as an instruction. Public web pages carry planted instructions. A fetch tool, a LlamaIndex retriever, a LangChain document loader, or an MCP response can carry the same payload.

The OWASP Top 10 for LLM Applications ranks prompt injection first and treats the indirect case as retrieved or tool-supplied content, not only a typed jailbreak. The lethal trifecta is why that content matters: private data plus untrusted text plus an egress tool is an exfiltration path.

A check on the original chat request doesn't see the page that the fetch tool just retrieved. That text never crossed the inbound HTTP body. The screen belongs inside the tool or retriever, on the string that you're about to return:

1
const content = await fetch(url).then((r) => r.text());
2

3
if (await isPromptInjection(content)) {
4
  return { content: "[Content blocked]" };
5
}
6

7
return { content };

On deny, return a placeholder. Don't pass the injected page back as model context. The same check belongs on a LlamaIndex retriever node and on a LangChain tool that loads URLs or tickets.

A second defense is the output shape. Trusted guidance (summary, suggestedActions) must never interpolate attacker-controlled paths, headers, or ticket bodies. Put raw evidence in an explicitly untrusted object. Schema descriptions that label the boundary help clients. They don't make the model safe. How we defend MCP tool outputs walks through that split. For the retriever version of the same split, see secure a RAG application, which screens each chunk before concatenation.

How do I prevent a malicious tool call from hijacking my AI agent?

Put a decision inside the tool handler, immediately before the side effect, and don't treat a clean inbound score as permission to send(), fetch(), or write. A hijack is usually a well-formed tool call the model was steered into, not a malformed argument a schema would reject.

Inbound screening asks whether this text should start a turn. It doesn't see sendEmail or create_pull_request. A framework allowlist and callbacks such as canUseTool do see the call and its arguments, but they are in-process client callbacks: the policy is whatever code you wrote there, with no centrally managed rules or organizational context behind it. The check that stops the hijack is the one that runs where the tool actually executes.

Use either of the following placements:

• Wrap the tool when the framework lets you insert a checkpoint between generated arguments and execute or invoke. On a policy denial the tool doesn't run, and the model gets a denial result that it can read.
• Call the check yourself when there's no adapter (the LlamaIndex case), or when you're screening tool output rather than arguments. The preceding fetch snippet is that path.

Decide what happens when the check can't finish. A search tool can fail open so that a timeout doesn't blank results. A Stripe refund must fail closed so that a timeout doesn't issue money. Choose per action.

A clean inbound score doesn't close a live email or Stripe tool. The send is still open. Sequence-aware deny (step 3 because of steps 1 and 2) is still hard; start by writing explicit checks for the dangerous combinations in your own app. For more information about that argument, see runtime controls on enterprise systems and anatomy of an agent incident.

How do AI security platforms detect prompt injection at runtime?

They use some mix of a specialist classifier or model, deterministic rules, and an LLM-as-judge evaluator. Those are different products. Only the ones that return a decision that your code acts on before the model or tool runs are enforcement.

Specialist classifier or model: a dedicated detector scores jailbreaks, role-play escapes, and instruction overrides. The application sends the text, gets a verdict, and blocks before the provider call. That costs a model call. Dry-run mode lets you measure false positives before you block. Whoever hosts the detector receives the prompt.

Rules: allow lists, deny lists, and checks on length and membership. They don't classify a jailbreak by meaning. They constrain what you can pass and which tool can run. They're useful as a second layer, not as the only screen.

LLM-as-judge: an evaluator model reads the prompt or tool call and returns a verdict. Datadog AI Guard is this shape: the application sends prompts or tool calls out, policy lives in the evaluator, and blocking depends on that remote judgment. Content leaves your environment for the evaluator to judge. Vendor trade-offs for that approach are in Datadog AI Guard vs Arcjet.

Sensitive-data inspection is a different control. Some platforms run it in your process so the raw body never leaves. Some send the body to a cloud scanner. Ask per control, not per logo. For more information about that question, see keeping security inspection local.

No detector catches every case. The hardest injections look like plausible business instructions and carry no attack pattern. Detection is one layer. Authorization at the action is the layer that still holds when the classifier misses. Pre-runtime vs post-runtime AI security has the test: if the tool still executes when the product "fires," it's detection, not enforcement.

Where should the check run in LangChain, LlamaIndex, and the Vercel AI SDK?

On the HTTP route for the user message, and inside every function that returns text to the model or creates a side effect.

LangChain and the Vercel AI SDK can wrap authored tools so that you don't hand-write the deny around every function. LlamaIndex apps do the same work inline. Don't invent a package that doesn't exist.

Rules that live next to the feature are reviewed in the same pull request. For more information about that placement argument, see enforce security rules at runtime in your code.