PII detection for production AI applications

What is PII detection for AI applications?

PII detection for AI applications is the practice of finding personally identifiable information in the text that flows through a model, and acting on it before that text is stored, logged, or sent to a provider. It covers the prompt a user submits, the reply the model returns, the arguments an agent passes to a tool, and the copies that land in your logs and your vector store.

The control that matters runs in your process and returns a decision, not a copy of the data. A scanner that ships the prompt to a second vendor to be classified has created another place the data lives. For a healthcare-specific version of this argument, see the best AI security for healthcare and regulated industries. For the field-level how-to on prompt and response inspection, see how to detect and redact PII in LLM inputs and outputs.

Where does PII enter and leave an AI application?

A model application leaks data at more points than the chat box. Each one needs its own check:

• The inbound prompt. A user pastes a card number, a medical detail, or a colleague's home address into a support form or a chat turn.
• The model reply. A model trained or retrieved on internal records can surface an employee's salary or a customer's account number in its answer.
• Tool call arguments. An agent that calls a payments or CRM tool passes fields you never see in the transcript.
• Retrieval context. A document pulled into a RAG prompt carries whatever PII the source held. See how to secure a RAG application.
• Logs, traces, and embeddings. These accumulate PII quietly, because nobody reads them until an incident.

Why does PII detection need to run in your process?

Where the inspection runs decides who else becomes a holder of the data. A cloud API that reads a prompt to classify it becomes a recipient of that prompt. A privacy review treats that vendor as a processor, and you inherit a data-processing agreement, a cross-border transfer question, and a second breach surface.

In-process detection avoids that. The analyzer runs inside your application, and only the result – the list of entity types that matched – leaves the function. The raw text stays where it started. This is the same reason to keep security inspection local: data residency is a property of where the bytes go, not of a checkbox in a contract.

How do you stop a user sending PII to a model?

Check the text on the way in, before you forward it to a provider. Arcjet's sensitiveInfo rule runs a local analyzer over a string and reports which entity types it found, so you can deny the turn or redact the field without the body leaving your process:

1
import arcjet, { sensitiveInfo } from "@arcjet/next";
2

3
const aj = arcjet({
4
  key: process.env.ARCJET_KEY!,
5
  rules: [
6
    sensitiveInfo({
7
      mode: "LIVE",
8
      deny: ["CREDIT_CARD_NUMBER", "EMAIL", "PHONE_NUMBER"],
9
    }),
10
  ],
11
});
12

13
export async function POST(req: Request) {
14
  const body = (await req.json()) as { message?: unknown };
15
  if (typeof body.message !== "string") {
16
    return new Response("Expected a message string.", { status: 400 });
17
  }
18
  const message = body.message;
19

20
  const decision = await aj.protect(req, { sensitiveInfoValue: message });
21
  if (decision.isDenied()) {
22
    return new Response("Remove personal details and try again.", {
23
      status: 400,
24
    });
25
  }
26
  // Safe to forward `message` to the model provider.
27
  return Response.json({ ok: true });
28
}

Keep the denial message generic, and don't echo the matched value back to the user. Start in DRY_RUN and read a real sample of what matches before you fail turns, so a legitimate message isn't blocked by a rule you never measured.

How do you stop a model surfacing confidential data?

The reply is untrusted too. A model with access to internal records can return an employee's or a customer's data in an answer, and a naive application streams that straight to the browser. Run the same check on the model output before you return it:

1
const completion = await model.generate(prompt);
2

3
const outbound = await aj.protect(req, {
4
  sensitiveInfoValue: completion.text,
5
});
6
if (outbound.isDenied()) {
7
  return new Response("The response was withheld for review.", {
8
    status: 502,
9
  });
10
}
11
return Response.json({ reply: completion.text });

Inspecting both directions closes the gap between "the user didn't send PII" and "the model didn't reveal any." Neither check ships the text to a third party.

Where does the check go in each framework?

The rule is the same everywhere. What changes is the object you already have in hand at the point you want to decide.

The middleware note is the one that costs people an afternoon. Putting the check in Next.js middleware feels right, because that's where a WAF-shaped control goes. Middleware doesn't have the parsed body, so the check either sees nothing or forces you to read the stream twice. Put it in the handler.

Where does detection sit in a latency budget?

Detection is cheap relative to what it protects, and the arithmetic is worth doing once so it stops being an objection.

Arcjet's bundled WebAssembly analysis adds under a millisecond. The optional on-device model that extends coverage to names, addresses, and government or financial identifiers adds roughly 6.6 ms median inference on Node.js, from a model of about 14.7 MB when 4-bit quantized. Controls that need cross-request state call the Arcjet Cloud API at typically 20 ms to 30 ms, with one call per decision regardless of how many rules ran.

Set that against the thing you're guarding: a model completion is measured in hundreds of milliseconds to seconds. Detection is a rounding error on the call it precedes.

Two places it stops being a rounding error, both worth planning for:

Scanning the whole thread on every turn. Detection cost is proportional to text length. If your client replays the full transcript and you scan all of it every time, cost grows with conversation length. Scan the latest message by default, and scan history only where history can still travel.

Fan-out in an agent loop. A check per tool call multiplies by the number of tool calls in a run. In-process detection stays small under that multiplication. A control that costs a network round trip per check doesn't.

Measure it in your own p99 rather than trusting either number. Time the handler with the rule and without it, on the same traffic, and look at the tail rather than the mean, because the tail is where a garbage collection pause meets a long message.

What happens when the detector is unavailable?

Every runtime control has to answer this, and "it won't happen" isn't the answer.

Two behaviors. Fail open means an unavailable check allows the request: the product stays up and the control is off. Fail closed means it refuses: the control holds and the feature is down.

Arcjet's direct guard() call fails open, returning an allow with error codes rather than throwing into your application. The server response has a timeoutSeconds deadline that defaults to 2, and a deadline produces a fail-open decision. That's the right default for a general-purpose control, and it's the wrong default for some of your routes, which is why the outcome is reported rather than hidden:

1
const decision = await arcjet.guard({
2
  label: "chat.user-message",
3
  actor: session.userId,
4
  rules: [pii(message)],
5
  // Lower this and you trade rule coverage for latency.
6
  timeoutSeconds: 1,
7
});
8

9
// Fail closed on this route: a timeout is not a clean result.
10
if (decision.conclusion === "DENY" || decision.hasFailedOpen()) {
11
  return refuseTurn();
12
}

Note what hasFailedOpen() describes: an outcome of this particular decision, not a configuration setting. It's true when the allow happened because something couldn't be processed.

Choose per action rather than globally. A recommendation feature and a route that sends data to a third party should not fail the same way.

Arcjet's engineering recommendation is fail open by default and fail closed on the specific routes where an unchecked operation is worse than a refused one, which is usually a small and enumerable list. The important part isn't which default you pick. It's that the choice is written down per route rather than inherited from whatever the library did.

Then test it. Block the Arcjet endpoint at the firewall in staging and exercise both a fail-open and a fail-closed route. A failure mode you've never run is a hypothesis.

How do you roll it out?

Three stages, and the temptation is to skip the middle one.

Stage 1, shadow. Every rule in DRY_RUN. Nothing is blocked. You're measuring what would have been. Run it long enough to include a weekend and a business-hours peak, because traffic shape changes what matches.

Stage 2, log-only with review. Still not enforcing, but now someone reads the detections weekly and classifies them. This is the stage that gets skipped, and it's the one that catches the legitimate business use of an entity you were about to block. A support product that collects phone numbers finds out here rather than in the support queue.

Stage 3, enforce, one class at a time. Move a single entity type from dry run to live. Watch the block rate and the session-completion rate for a week. Then the next one. Moving five classes at once means a regression you can't attribute.

Two things to have in place before stage 3. A way to turn a rule off without a deploy, because the first bad enforcement decision happens at an inconvenient hour. And an alert on block-rate change rather than on block count, since a rule that starts firing ten times more often has either found an attack or broken a legitimate flow, and you want to know within the hour either way.

Keep a permanent dry-run tier after rollout. The classes you chose not to enforce still tell you what your traffic contains, and that's what tells you when to enforce the next one.

What should a PII detection tool for AI applications do?

The useful evaluation is per capability, not per logo. A tool that fits a production AI application does the following:

• Runs in the request path. It returns an allow, deny, or redact decision fast enough to keep in the handler, not an asynchronous report you read after the data has already gone to a provider.
• Inspects in your process. The text being classified stays inside your environment, and only the matched entity types leave.
• Covers input, output, and free-text tool arguments. A check on the chat box alone misses the reply and the tool call.
• Names what it cannot detect. A tool that silently misses government IDs or names is worse than one that requires you to configure them, because you build on an assumption that fails in production.
• Supports dry-run. You measure the false-positive rate on real traffic before you block anyone.

Arcjet's bundled analyzer detects email address, phone number, IP address, and credit card number locally. Names, government IDs, and addresses need a custom detection function or an additional backend, and asking for them without one is a startup configuration error rather than a silent miss.

What does a security review ask, and what are the answers?

The questions come in a predictable order. Having the answers written down turns a two-week review into a meeting.

Where is the text classified? In your process, in a WebAssembly module bundled with the SDK. The raw text is not transmitted.

What leaves your environment? The decision record: the entity types that matched, the rule label and mode, an actor identifier, and a SHA-256 hash of the scanned text. Say the hash out loud rather than omitting it, because a reviewer who finds it later will assume you were hiding it. A hash of a low-entropy value is guessable, so treat it as an identifier rather than as anonymized data.

Which controls do transmit content? Prompt-injection detection sends the prompt, because a specialist model makes that decision. Bot protection and rate limiting send signals and keys rather than bodies. The control-by-control table is the answer to this question in full.

What happens if the vendor is unavailable? Detection is in-process, so classification doesn't depend on the network. The decision call fails open by default within a configurable timeout, and the outcome is reported on the decision so a route can fail closed instead. Name the routes that do.

What is retained, where, and for how long? Decision records, not content. Point at the retention documentation rather than paraphrasing it.

How do you know it works? Dry-run measurements on your own traffic, per entity type, with the false-positive review that came out of stage 2.

What does it not detect? The bundled analyzer covers email addresses, phone numbers, IP addresses, and credit card numbers. Names, addresses, and government or financial identifiers need the on-device model backend or a custom detection function. Naming an entity type the active backend can't emit is a configuration error, so the rule refuses to build rather than matching nothing forever. That is the behavior you want: a detector that finds nothing is indistinguishable from a clean scan, and you would build on it.

Is it a data discovery tool? No. It decides whether this content should move, at this boundary. It doesn't map where regulated data lives across your systems.

How does PII detection relate to GDPR and CCPA?

GDPR and CCPA both turn on where personal data goes and who processes it. Sending a prompt that contains an EU resident's data to a model provider, and to a separate classification API, adds processors and transfers you then have to document and justify. Detecting and redacting the data in your process before either call reduces the set of parties that ever hold it, which is the outcome both regimes reward.

In-process detection is a control, not a compliance certificate. It helps you honor data-minimization and purpose-limitation, and it produces a decision you can log for an audit. It does not by itself make an application GDPR or CCPA compliant, and any vendor that says a single rule does is selling an adjective. Pair detection with retention limits, access controls, and a data-subject-request process.

What should you do this week?

1. List the points where PII enters and leaves your AI application: the prompt, the reply, tool arguments, retrieval context, and logs.
2. Put a sensitiveInfo check on the inbound route in DRY_RUN, and read a real sample of what it matches.
3. Add the same check on the model output before you return it.
4. Configure custom detection for the entity types your application handles that the bundled analyzer doesn't cover, and fail startup if they're missing.
5. Redact PII in your own logs and traces so a value that arrives as a field isn't archived as a string. See how to redact sensitive data from Go logs.
6. Switch the rules from DRY_RUN to LIVE once the sample is clean.

The data you never forwarded is the data you never have to explain.