How to detect and redact PII in LLM inputs and outputs

How do I detect and redact PII in LLM inputs and outputs?

Scan the text in your own process at two boundaries: the prompt before it goes to the provider, and the completion before it reaches the user. Then act on the result. Block when the data should never have been there, redact when the turn should continue without the value, and log without enforcing while you're still measuring.

Both halves matter, and most implementations have one. Inbound personally identifiable information (PII) is the string in the chat box: a card, or a resume that the user wants rewritten. It hasn't left your app yet, and the next hop is OpenAI or Anthropic. Outbound PII is the model repeating something it was given, or a tool result traveling further than it should.

A support bot took a card paste. The model said it couldn't take payment here. Privacy still opened a ticket, because the provider prompt log already had the digits. Most teams scan the reply. The prompt is the one that leaves first.

The first pull request for a chat route usually looks like this:

1
export async function POST(req: Request) {
2
  const { message } = await req.json();
3

4
  const reply = await callProvider({
5
    messages: [{ role: "user", content: message }],
6
  });
7

8
  return Response.json({ reply });
9
}

There's no scan. The user message goes straight into the provider call. Paste 4242 4242 4242 4242 into the box. The provider has the card before your handler looks at reply. A regex on the completion doesn't unwind that. A "we do not train on your data" checkbox doesn't unwind it either. The string left.

Detect means you found a span. Then you pick what happens to the turn. A finding doesn't stop the call.

How do you check the prompt before the provider call?

Put the check in the handler, after the body is parsed and before the provider client is touched.

In the Next.js App Router that's the route handler or a server action. Not middleware: middleware runs before the body is read, so it can't see the text you need.

1
import arcjet, { sensitiveInfo } from "@arcjet/next";
2

3
const aj = arcjet({
4
  key: process.env.ARCJET_KEY!,
5
  rules: [
6
    sensitiveInfo({
7
      mode: "LIVE",
8
      deny: ["CREDIT_CARD_NUMBER"],
9
    }),
10
  ],
11
});
12

13
export async function POST(req: Request) {
14
  const body = (await req.json()) as { message?: unknown };
15
  if (typeof body.message !== "string") {
16
    return new Response("Expected a message string.", { status: 400 });
17
  }
18

19
  const decision = await aj.protect(req, {
20
    sensitiveInfoValue: body.message,
21
  });
22

23
  if (decision.isDenied()) {
24
    return new Response("Please rephrase your message.", { status: 400 });
25
  }
26

27
  const reply = await callProvider({
28
    messages: [{ role: "user", content: body.message }],
29
  });
30

31
  return Response.json({ reply });
32
}

On Express the package is @arcjet/node and the check goes after the body parser, for the same reason:

1
import arcjet, { sensitiveInfo } from "@arcjet/node";
2
import express from "express";
3

4
const app = express();
5
app.use(express.json());
6

7
const aj = arcjet({
8
  key: process.env.ARCJET_KEY!,
9
  rules: [
10
    sensitiveInfo({
11
      mode: "LIVE",
12
      deny: ["CREDIT_CARD_NUMBER"],
13
    }),
14
  ],
15
});
16

17
app.post("/chat", async (req, res) => {
18
  const decision = await aj.protect(req, {
19
    sensitiveInfoValue: req.body.message,
20
  });
21

22
  if (decision.isDenied()) {
23
    res.status(400).json({ error: "Please rephrase your message." });
24
    return;
25
  }
26

27
  res.json({ reply: await callProvider(req.body.message) });
28
});

Keep the deny response generic. "Please rephrase your message." Don't name the entity. Don't say you found a card. The user who is probing will use that.

On a path with no HTTP request, the same check comes from @arcjet/guard, which takes the string directly:

1
import { launchArcjet, localDetectSensitiveInfo } from "@arcjet/guard";
2

3
const arcjet = launchArcjet({ key: process.env.ARCJET_KEY! });
4
const pii = localDetectSensitiveInfo({ deny: ["CREDIT_CARD_NUMBER"] });
5

6
const decision = await arcjet.guard({
7
  label: "tools.summarize-ticket",
8
  actor: session.userId,
9
  correlationId: runId,
10
  rules: [pii(ticketBody)],
11
});

Detection runs in a WebAssembly module inside your process either way, so the raw body isn't transmitted to be classified. For the residency argument in full, see keeping security inspection local.

Regex, classifier, or both?

The three approaches have different failure modes, and the useful answer is which entity types each one is right for.

Hybrid is what a production system ends up with, and it's worth being explicit about why rather than treating it as a feature list. A card number either passes a Luhn check or it doesn't, and no model beats arithmetic on that. A name is a name because of the sentence around it, and no pattern captures that.

Arcjet's bundled analyzer covers the structured types locally: email addresses, phone numbers, IP addresses, and credit card numbers. The optional on-device model extends that to names, addresses, and government and financial identifiers, adding roughly 6.6 ms median inference on Node.js from a model of about 14.7 MB when 4-bit quantized. Both run in your process.

The extended types need the backend passed explicitly, and this is the detail that most often produces a rule that finds nothing:

1
import arcjet, { sensitiveInfo } from "@arcjet/next";
2
import { rampart } from "@arcjet/sensitive-info-rampart";
3

4
const aj = arcjet({
5
  key: process.env.ARCJET_KEY!,
6
  rules: [
7
    sensitiveInfo({
8
      mode: "LIVE",
9
      // SSN needs the on-device model. Without `backend`, this config errors.
10
      deny: ["CREDIT_CARD_NUMBER", "SSN"],
11
      backend: rampart(),
12
    }),
13
  ],
14
});

On a Guard client the equivalent is sensitiveInfoBackend: rampart() passed to launchArcjet. Either way, naming an entity type the active backend can't emit is a configuration error rather than a rule that quietly matches nothing. The rule refuses to build, so a missing backend shows up at startup instead of as a clean scan you would have trusted.

Whichever combination you land on, measure it on your own traffic in dry run before enforcing. Vendor accuracy numbers are measured on a corpus the vendor picked, and yours is different.

How do you detect your own identifiers?

Every application has identifiers that no general detector knows: an internal account reference, a medical record number, an employee identifier, a policy number with a format that exists only in your company.

The sensitiveInfo rule takes a detect function for these. It receives the tokenized input and returns an entity type or undefined for each token, so a custom type sits alongside the built-in ones in the same deny list:

1
import arcjet, { sensitiveInfo } from "@arcjet/next";
2

3
/** Internal account references look like ACC-000000. */
4
function detectAccountRef(tokens: string[]) {
5
  return tokens.map((token) =>
6
    /^ACC-\d{6}$/.test(token) ? "ACCOUNT_REF" : undefined,
7
  );
8
}
9

10
const aj = arcjet({
11
  key: process.env.ARCJET_KEY!,
12
  rules: [
13
    sensitiveInfo({
14
      mode: "LIVE",
15
      deny: ["CREDIT_CARD_NUMBER", "ACCOUNT_REF"],
16
      detect: detectAccountRef,
17
    }),
18
  ],
19
});

Two mechanics to get right. The returned array has to be the same length as the token array, one entry per token, or the mapping between findings and positions breaks. And an identifier that spans multiple tokens needs contextWindowSize raised so the function sees enough neighboring tokens to recognize it, since the default window is narrow.

Custom entity types are a feature of the sensitiveInfo rule in the HTTP SDKs. @arcjet/guard doesn't support them; on that path you write a custom rule instead. Worth knowing before you design around it.

Keep the function cheap. It runs on every token of every message, so a regular expression is fine and a database lookup isn't.

How do you check the completion before it renders?

The output path is a different job with a different deny list, and it's the half that usually goes missing.

A model can reproduce a value that arrived through context: from the user two turns ago, from a retrieved document, from a tool result. It can also produce something that looks like a card number without one ever having been present. Both are reasons to check.

1
const completion = await callProvider(messages);
2

3
const outbound = await aj.protect(req, {
4
  sensitiveInfoValue: completion.text,
5
});
6

7
if (outbound.isDenied()) {
8
  return new Response("The response was withheld for review.", {
9
    status: 502,
10
  });
11
}
12

13
return Response.json({ reply: completion.text });

The outbound list should differ from the inbound one. Inbound, an email address is often legitimate: the user is telling you where to send a confirmation. Outbound, the same email address in a completion may mean the model just repeated another customer's record back to this one.

Check before you render, and check before you persist. The rendering check catches what the user sees. It misses the copy in the log, the copy in the trace, and the copy written to a vector store, and those outlive the conversation by months.

What changes when the response streams?

The input side doesn't change. You already scanned before the connection opened.

The output side does, because a detected span can straddle a chunk boundary. 4242 4242 arrives in one chunk and 4242 4242 in the next, and a per-chunk scan sees two harmless number pairs.

Three options:

Buffer a tail. Hold back the last N characters rather than emitting them immediately, and scan across the join. N has to exceed the longest entity you're looking for. The user still sees streaming; the final fragment is one chunk behind.

Scan a sliding window. Keep a rolling buffer of recent chunks, run detection across the window, and emit what's behind it. More robust for long entities, slightly more code.

Buffer the whole completion. Correct, simple, and it throws away the reason you were streaming. Fine for a short structured response, wrong for long-form output.

The decision rule is straightforward. If the entity types you're scanning for outbound are ones that must never render, buffer, because streaming is a UX preference and that isn't. If you're scanning to log rather than to block, per-chunk with a sliding window is enough.

Do you scan the latest message or the whole thread?

Scan the latest user message by default. Scan the thread when earlier turns still travel.

A card arrives on Tuesday and you block the turn. On Wednesday the client sends the full transcript with every request, as most chat clients do, and Tuesday's paste rides along inside messages while today's message is clean. The block fired once. The string has gone to the provider on every turn since.

If your client replays history, either scan the payload you're about to send rather than the message you just received, or strip denied turns from the stored transcript at the moment you block them. Blocking a turn and then storing it is the common shape of this bug.

Cost scales with text length, so scanning a long transcript on every turn is real work. Scanning what you're about to send is both the correct answer and the cheaper one, because you were going to pay to transmit it anyway.

When do you redact instead of blocking?

Redact when the turn should continue without the value. A local redact step returns the stripped string and a function that restores the originals:

1
import { redact } from "@arcjet/redact";
2

3
const [redacted, unredact] = await redact(message, {
4
  entities: ["email", "credit-card"],
5
});
6

7
const completion = await callProvider({
8
  messages: [{ role: "user", content: redacted }],
9
});
10

11
// Only restore where the interface needs the original value.
12
const reply = unredact(completion);

The provider sees a placeholder such as <Redacted email #1>, stable within the call, so the model can reason about the entity without holding it. Restore on the way back for the human reading the reply, and not for a log, a trace, an embedding, or another provider call.

Redaction with no deny strips the span and lets the turn continue, which is right for a resume-rewriting box and wrong for a payments bot. If the policy is that a field must never reach a model, that's a block, not a placeholder. For provider-specific middleware, streaming, and reversible versus irreversible replacement, see how to redact sensitive data before sending it to OpenAI or Anthropic.

How do you run this in production?

Dry run first, on real traffic, for long enough to include a weekend. Read the detections and classify them before you enforce anything, because that's where you find the legitimate business use of an entity you were about to block. Then move one entity class at a time from dry run to live.

Decide separately what happens when the detector doesn't answer. A timeout is not a clean score, and treating it as one means the control disappears during exactly the incident you'd want it for.

For staged rollout, fail-open versus fail-closed per route, latency budgets, per-framework placement, and the questions a security review will ask, see PII detection for production AI applications.

How do I stop users from sending PII to an LLM?

Put the check on the inbound message before the provider call, and choose per data class whether you block, redact, or only warn. A terms-of-service checkbox isn't a control, a system prompt line saying "do not accept card numbers" isn't a control, and a client-side regex is a usability feature.

The part that's easy to get wrong is logging the block. Record the entity types that matched, never the matched span, or the detector becomes the leak. For the policy table, the shadow AI problem, and how to log a block without logging the PII, see how to stop users sending PII to an LLM.

Which tools prevent PII leakage from AI agents?

The split is where the body goes to get classified, and whether the check can run on a path that has no HTTP request. In-process detection classifies the string where it already is. Cloud DLP receives it. Self-hosted scanners keep it in your environment at the cost of a service to run. Provider-side filters run after the bytes crossed the wire. Edge proxies never see a tool call at all.

An agent has five leak points rather than two, and tool call arguments are the one most products miss. For the coverage matrix, see how to prevent PII leakage from AI agents, and for the platform-by-platform version, see PII detection at runtime: gateway, sidecar, or in-process.

How does this relate to GDPR and CCPA?

Local inspection shrinks the set of vendors that receive the prompt, because the content is never disclosed to a scanner. That removes a transfer question rather than answering one, and it doesn't make an application compliant. No tool does.

Redaction before transmission maps directly to GDPR Article 5(1)(c) data minimization. It doesn't end your processor relationship with the model provider, which is a separate obligation with its own contract and transfer mechanism. For the requirement-by-requirement mapping, including erasure across derived stores and audit trails that don't create a second PII store, see GDPR and CCPA compliance for LLM applications.

Does a clean PII score authorize a tool call?

No. A clean score means the prompt didn't contain a card, an email, a phone, or an IP that you deny. It doesn't mean this queryCustomer is the right row for the user in session. For more information about authorization at access, see how to stop AI agents accessing data they should not.

A clean PII score isn't a jailbreak check either. Run prompt injection detection on the same text; it answers a different question. Tool results, logs, and embeddings belong to the data exfiltration job, and a model surfacing records it shouldn't have retrieved belongs to confidential data disclosure.

The ticket is a screenshot of the last user bubble. Sixteen digits, grouped the way people read a Visa. The model reply was polite and empty. Scan the bubble, and scan the reply.