How to detect and redact PII in LLM inputs and outputs

How do I detect and redact PII in LLM inputs and outputs?

Inbound personally identifiable information (PII) is the string in the chat box: a card, or a resume that the user wants rewritten. It hasn't left your app yet. The next hop is OpenAI or Anthropic.

Outbound PII is a different job. A tool already ran. A field is sitting in a result, a log line, a trace, or an embedding. For more information about that path, see how to prevent data exfiltration through AI agents. This page stops at the prompt and the completion.

A support bot took a card paste. The model said it couldn't take payment here. Privacy still opened a ticket, because the provider prompt log already had the digits. Most teams scan the reply. The prompt is the one that leaves first.

The first pull request for a chat route usually looks like this:

1
export async function POST(req: Request) {
2
  const { message } = await req.json();
3

4
  const reply = await callProvider({
5
    messages: [{ role: "user", content: message }],
6
  });
7

8
  return Response.json({ reply });
9
}

There's no scan. The user message goes straight into the provider call. Paste 4242 4242 4242 4242 into the box. The provider has the card before your handler looks at reply. A regex on the completion doesn't unwind that. A "we do not train on your data" checkbox doesn't unwind it either. The string left.

Detect means you found a span. Then you pick what happens to the turn. A finding doesn't stop the call.

Block when the data must never reach the model and failing the turn is fine. A payments bot that must not take a card in chat is a block. Redact when the turn can continue without the span; the provider sees a placeholder. Flag when you're still measuring. That's dry run.

Start with structured types: email, phone, IP, and card numbers with a Luhn check so 4242424242424241 doesn't fire. Names, street addresses, and government or financial IDs need a stronger detector. Tune the list to the product. A support bot that legitimately collects a phone number must not fail the turn for a phone.

Run the detector in your process so that the raw body doesn't have to visit a second vendor to get classified. For more information about that argument, see keeping security inspection local.

Other local detectors plug in at the same place: scan message, then decide, then call the model. The following example runs an in-process scan before the provider call with Arcjet sensitive information detection:

1
import arcjet, { sensitiveInfo } from "@arcjet/next";
2

3
const aj = arcjet({
4
  key: process.env.ARCJET_KEY!,
5
  rules: [
6
    sensitiveInfo({
7
      mode: "LIVE",
8
      deny: ["CREDIT_CARD_NUMBER", "EMAIL"],
9
    }),
10
  ],
11
});
12

13
const decision = await aj.protect(req, {
14
  sensitiveInfoValue: message,
15
});
16

17
if (decision.isDenied()) {
18
  return new Response("Please rephrase your message.", { status: 400 });
19
}

Keep the deny response generic. "Please rephrase your message." Don't name the entity. Don't say you found a card. The user who is probing will use that.

After the provider returns, run the same check on the completion before you render it. The prompt check is the one that stops the hop. The completion check is how you keep a reproduced card out of the browser.

A card isn't a jailbreak. A jailbreak isn't a card. Runtime security for LLM applications already splits those jobs. For rule wiring, see the Arcjet data loss prevention docs and the AI agent security platform comparison.

PII detection for production AI applications

Production isn't a regex on the last bubble. It's a deny list that you measured on real prompts, and a choice about whether today's message or the whole thread gets scanned.

Dry run first. Log detections without failing turns. Run that against real prompts until you've looked at a sample. You'll see email from people typing their address into a "where's my order" prompt. You'll see cards from the paste. You'll also see the misses, like a name or a street address. Don't start failing turns until you've decided which entities fail the turn. Then enforce.

Pass the most recent user message. That's the usual case. Pass the thread if yesterday's paste is still in context. A card from Tuesday rides along in messages even if today's bubble is clean. A "safe" follow-up ("can you try again?") replays the original paste if the client sends the whole transcript.

Keep denied responses generic on every path. Don't return CREDIT_CARD_NUMBER, a highlighted span, or an explanation that a detector fired.

The same check belongs on a tool, an MCP handler, or a queue job. There's no HTTP request on those paths. The text is still inbound, and you still have to scan it before the provider. If the detector errors, decide what the job does. A timeout isn't a clean score.

How to redact sensitive data before sending to OpenAI or Anthropic

Block when failing the turn is the point. Redact when the model needs only a placeholder.

A redact step returns two things – the stripped string, and a function that puts the originals back:

1
const [redacted, unredact] = await redact(message, {
2
  entities: ["email", "credit-card"],
3
});
4

5
const completion = await callProvider({
6
  messages: [{ role: "user", content: redacted }],
7
});
8

9
// Only unredact if the product needs the original in the UI
10
const reply = unredact(completion);

The provider sees a placeholder such as <Redacted email #1>. If the bot is supposed to say you have the email on file, that's enough. Unredact on the way back when the UI needs the original, not because the model asked for it.

Redact alone, with no deny, strips the span and lets the turn continue. That's fine for a rewrite-my-resume box. It's the wrong control for a payments bot. If the policy is that this field must never go to a model, you need a deny, not a placeholder.

The same sequence belongs on retrieved context. Your application trusts a RAG chunk by default, which makes it a common leak path. Inspect the chunk before it becomes model context. For more information about that boundary, see how to secure a RAG application. Apply the same detection to logs, traces, and embeddings. Those copies live longer than the request that produced them.

Sending the prompt to a cloud scanner to decide whether it's safe for OpenAI or Anthropic doesn't reduce the number of third parties that see it.

How do I stop users from sending PII to an LLM?

Put a check on the inbound message before the provider call. Once the string is in the provider prompt log, you're doing incident response.

Users still try. The control isn't a terms-of-service checkbox, and it isn't a system-prompt line that says "do not accept card numbers." The model isn't the enforcement point. The scan in front of the provider call is.

The same inbound check doesn't replace an output check. A model can reproduce a card that arrived through context, or a tool result that you already fetched. Scan the completion before you render it. Scan what a tool is about to return before it travels. For more information about that second job, see preventing data exfiltration through AI agents.

Best tools to prevent PII leakage from AI agents

The split is where the body goes to get classified, and whether the check can run on a path that has no HTTP request.

Local in-process. The detector runs in your application. The raw prompt never leaves your application for classification. That's the only shape that doesn't add a recipient of the data that you're trying not to send. It has to work on HTTP chat routes and on tools, MCP handlers, and queue jobs.

Cloud DLP. Cloud data loss prevention (DLP) products classify well. The scanner receives the body. Privacy review says so. You've built a second export path to prevent the first. For more information about the scanner problem, see preventing data exfiltration through AI agents.

Self-hosted scanners. Microsoft Presidio classifies well too, and it's open source and runs in your own environment, so the body doesn't leave. The trade is operational: in-process use means Python, and everywhere else it's a separate service that you deploy and keep running. A JavaScript tool handler or queue job reaches it over the network, not in-process.

Provider-side filters. A provider's own safety filters or a "do not log this" flag run after the string has left your app. Too late for the hop you care about. The OpenAI moderation API isn't in this list at all: it classifies content-safety categories, not PII. A provider promise not to train on the data doesn't unwind a prompt log that you didn't mean to create.

Remote evaluators. Some platforms send the prompt or tool call to a vendor for a verdict, including as telemetry. Datadog AI Guard is that shape. If the content carries a residency or minimization obligation, there's a transfer to review.

Sidecars without a PII explainer. A sidecar around the agent runtime can see actions. That isn't the same as a documented inbound PII detector. Rein doesn't publish a PII explainer. Don't treat a missing explainer as a hidden PII product.

Regex isn't a tool in this list. It misses a formatted card (4242 4242 4242 4242 versus 16 digits in a row). It never finds a name. Teams ship /\d{16}/ as PII detection and then discover the paste with spaces.

Pick the tool that can run before the provider call, on HTTP and on tools, without making a new processor of the body that you're trying not to send. For vendor-by-vendor specifics, see the AI agent security platform comparison.

How do I comply with GDPR and CCPA when using LLMs?

Local inspection shrinks the set of vendors that receive the prompt. It doesn't make you compliant. No tool does.

Under GDPR, a vendor that processes personal data on your behalf is a processor; some recipients act as independent controllers instead. A processor relationship needs an Article 28 agreement and sub-processor transparency. Any international transfer needs a Chapter V mechanism, such as an adequacy decision or standard contractual clauses. Inspection that runs in-process doesn't create that relationship for the content inspected, because the content is never disclosed. That removes a transfer question rather than answering it.

Data minimization is the other principle that maps directly. Article 5(1)(c) requires that processing be limited to what is necessary. When an in-process equivalent exists, exporting full request bodies to a third party to detect personal data is difficult to defend as minimal.

CCPA and CPRA turn on similar structure: whether a disclosure to a service provider has occurred, and what contractual terms govern it. Fewer recipients is a simpler argument in both regimes.

This isn't legal advice, and the analysis depends on your own processing. Local PII detection doesn't replace a lawful basis, a retention schedule, a records-of-processing entry, or a vendor review of OpenAI or Anthropic. It changes which questions you have to answer about whoever inspects the prompt. For more information about where inspection happens and what that implies for residency, see keeping security inspection local.

Does a clean PII score authorize a tool call?

No. A clean score means the prompt didn't contain a card, an email, a phone, or an IP that you deny. It doesn't mean this queryCustomer is the right row for the user in session. For more information about authorization at access, see how to stop AI agents accessing data they should not.

A clean PII score isn't a jailbreak check. Run prompt injection on the same text. Tool results, logs, and embeddings belong to the data exfiltration job. A clean score on the last user bubble doesn't authorize a refund, a CRM read, or a send.

The ticket is a screenshot of the last user bubble. Sixteen digits, grouped the way people read a Visa. The model reply was polite and empty. Scan the bubble.