PII detection for AI applications

What is PII detection for AI applications?

PII detection for AI applications is the practice of finding personally identifiable information in the text that flows through a model, and acting on it before that text is stored, logged, or sent to a provider. It covers the prompt a user submits, the reply the model returns, the arguments an agent passes to a tool, and the copies that land in your logs and your vector store.

The control that matters runs in your process and returns a decision, not a copy of the data. A scanner that ships the prompt to a second vendor to be classified has created another place the data lives. For a healthcare-specific version of this argument, see the best AI security for healthcare and regulated industries. For the field-level how-to on prompt and response inspection, see how to detect and redact PII in LLM inputs and outputs.

Where does PII enter and leave an AI application?

A model application leaks data at more points than the chat box. Each one needs its own check:

• The inbound prompt. A user pastes a card number, a medical detail, or a colleague's home address into a support form or a chat turn.
• The model reply. A model trained or retrieved on internal records can surface an employee's salary or a customer's account number in its answer.
• Tool call arguments. An agent that calls a payments or CRM tool passes fields you never see in the transcript.
• Retrieval context. A document pulled into a RAG prompt carries whatever PII the source held. See how to secure a RAG application.
• Logs, traces, and embeddings. These accumulate PII quietly, because nobody reads them until an incident.

Why does PII detection need to run in your process?

Where the inspection runs decides who else becomes a holder of the data. A cloud API that reads a prompt to classify it becomes a recipient of that prompt. A privacy review treats that vendor as a processor, and you inherit a data-processing agreement, a cross-border transfer question, and a second breach surface.

In-process detection avoids that. The analyzer runs inside your application, and only the result – the list of entity types that matched – leaves the function. The raw text stays where it started. This is the same reason to keep security inspection local: data residency is a property of where the bytes go, not of a checkbox in a contract.

How do you stop a user sending PII to a model?

Check the text on the way in, before you forward it to a provider. Arcjet's sensitiveInfo rule runs a local analyzer over a string and reports which entity types it found, so you can deny the turn or redact the field without the body leaving your process:

1
import arcjet, { sensitiveInfo } from "@arcjet/next";
2

3
const aj = arcjet({
4
  key: process.env.ARCJET_KEY!,
5
  rules: [
6
    sensitiveInfo({
7
      mode: "LIVE",
8
      deny: ["CREDIT_CARD_NUMBER", "EMAIL", "PHONE_NUMBER"],
9
    }),
10
  ],
11
});
12

13
export async function POST(req: Request) {
14
  const body = (await req.json()) as { message?: unknown };
15
  if (typeof body.message !== "string") {
16
    return new Response("Expected a message string.", { status: 400 });
17
  }
18
  const message = body.message;
19

20
  const decision = await aj.protect(req, { sensitiveInfoValue: message });
21
  if (decision.isDenied()) {
22
    return new Response("Remove personal details and try again.", {
23
      status: 400,
24
    });
25
  }
26
  // Safe to forward `message` to the model provider.
27
  return Response.json({ ok: true });
28
}

Keep the denial message generic, and don't echo the matched value back to the user. Start in DRY_RUN and read a real sample of what matches before you fail turns, so a legitimate message isn't blocked by a rule you never measured.

How do you stop a model surfacing confidential data?

The reply is untrusted too. A model with access to internal records can return an employee's or a customer's data in an answer, and a naive application streams that straight to the browser. Run the same check on the model output before you return it:

1
const completion = await model.generate(prompt);
2

3
const outbound = await aj.protect(req, {
4
  sensitiveInfoValue: completion.text,
5
});
6
if (outbound.isDenied()) {
7
  return new Response("The response was withheld for review.", {
8
    status: 502,
9
  });
10
}
11
return Response.json({ reply: completion.text });

Inspecting both directions closes the gap between "the user didn't send PII" and "the model didn't reveal any." Neither check ships the text to a third party.

What should a PII detection tool for AI applications do?

The useful evaluation is per capability, not per logo. A tool that fits a production AI application does the following:

• Runs in the request path. It returns an allow, deny, or redact decision fast enough to keep in the handler, not an asynchronous report you read after the data has already gone to a provider.
• Inspects in your process. The text being classified stays inside your environment, and only the matched entity types leave.
• Covers input, output, and free-text tool arguments. A check on the chat box alone misses the reply and the tool call.
• Names what it cannot detect. A tool that silently misses government IDs or names is worse than one that requires you to configure them, because you build on an assumption that fails in production.
• Supports dry-run. You measure the false-positive rate on real traffic before you block anyone.

Arcjet's bundled analyzer detects email address, phone number, IP address, and credit card number locally. Names, government IDs, and addresses need a custom detection function or an additional backend, and asking for them without one is a startup configuration error rather than a silent miss.

How does PII detection relate to GDPR and CCPA?

GDPR and CCPA both turn on where personal data goes and who processes it. Sending a prompt that contains an EU resident's data to a model provider, and to a separate classification API, adds processors and transfers you then have to document and justify. Detecting and redacting the data in your process before either call reduces the set of parties that ever hold it, which is the outcome both regimes reward.

In-process detection is a control, not a compliance certificate. It helps you honor data-minimization and purpose-limitation, and it produces a decision you can log for an audit. It does not by itself make an application GDPR or CCPA compliant, and any vendor that says a single rule does is selling an adjective. Pair detection with retention limits, access controls, and a data-subject-request process.

What should you do this week?

1. List the points where PII enters and leaves your AI application: the prompt, the reply, tool arguments, retrieval context, and logs.
2. Put a sensitiveInfo check on the inbound route in DRY_RUN, and read a real sample of what it matches.
3. Add the same check on the model output before you return it.
4. Configure custom detection for the entity types your application handles that the bundled analyzer doesn't cover, and fail startup if they're missing.
5. Redact PII in your own logs and traces so a value that arrives as a field isn't archived as a string. See how to redact sensitive data from Go logs.
6. Switch the rules from DRY_RUN to LIVE once the sample is clean.

The data you never forwarded is the data you never have to explain.