Content moderation for AI applications

How do I add content moderation to an AI application?

Screen the string that the user typed before you send it to a model. Then screen any text that the model returns before you persist it, render it, email it, or charge for it. The framework changes the handler. It doesn't change the hop.

An image generator, a chat box, and a "rewrite this listing" form share one shape. The user supplies a prompt. Your app calls a provider. Generated content comes back. A line in the system prompt that says "refuse harmful requests" isn't a deny. The model isn't the enforcement point.

Content moderation is the check that answers one question about a piece of text: is this harmful to store, show, or forward? This page covers the architecture and the requirements for that check on a free-text field. Content moderation isn't personally identifiable information (PII) detection, and it isn't prompt injection protection. Those are different detectors on the same string.

Where does the check sit when users supply prompts for generated content?

Put the check on the string that you're about to lose control of. Skip session IDs and message UUIDs. Don't scan the whole JSON body.

Place the check on each of the following surfaces:

An image prompt is still text. The user typed a description. That description is the hop that you own. Arcjet content moderation evaluates untrusted text. It doesn't classify image bytes. Provider-side image filters are a separate control. If you also write a caption or alternative text, then that caption is another text hop.

A clean inbound score doesn't unwind a completion that you already sent. A clean completion doesn't unwind a prompt that already sat in a provider log. Run the check on each hop that you take.

If the product charges for a generation, then the deny sits in front of the charge. A later refund doesn't rewind the fact that the charge existed.

How do I moderate an image prompt?

Extract the prompt string. Run the harmful-content check. Call images.generate, or the equivalent, only after an allow. The following handler screens the prompt, then generates an image:

1
export async function POST(req: Request) {
2
  const { prompt } = await req.json();
3

4
  if (typeof prompt !== "string" || prompt.length === 0) {
5
    return new Response("Bad request.", { status: 400 });
6
  }
7

8
  if (await isHarmful(prompt)) {
9
    return new Response("Please rephrase your prompt.", { status: 400 });
10
  }
11

12
  const image = await createImage(prompt);
13
  return Response.json({ image });
14
}

isHarmful is the detector that you already call, and createImage is your own call to the image provider. The shape is the same on a FastAPI route. The generated file is out of scope for a text check. If you later ask a model to write a title for that image, then screen that title before you store or display it.

What does a production content-moderation check have to do?

A detector that only logs isn't enforcement. Production means the handler returns before the next hop. A production check does all of the following:

• Read the free-text field, not the whole request.
• Return a decision that your code acts on: allow or deny.
• Fail the hop on deny: no provider call, no persist, no render, no forward, no charge.
• Keep the user-facing body generic. Use one inbound sentence and one outbound sentence.
• Dry run on real traffic first. Staging traffic doesn't include the prompts that you see in production.
• Decide what happens if the detector errors. A timeout isn't a clean score. A marketing FAQ can keep serving. A paid generation must fail closed.

Don't return a category, a score, or the flagged span. The person who is probing will use that. If you also run PII or prompt-injection checks on the same string, then share the deny body and split only the logs.

How is this different from PII detection and prompt injection?

Content moderation, prompt injection, and PII detection ask three questions on one string:

• Content moderation asks whether the text is harmful to store, show, or forward.
• Prompt injection asks whether the text is trying to instruct the model.
• PII detection asks whether the string contains a denied entity, such as a card.

A jailbreak can be non-toxic. A card paste can be polite. A harmful image prompt can contain no card and no instruction override. A clean score on one isn't a clean score on the others. The OWASP Top 10 for LLM Applications ranks prompt injection first, and it treats that risk separately from the harmful-content question that this page covers.

How to detect and redact PII in LLM inputs and outputs already says that the OpenAI Moderation API isn't a PII detector. That API classifies content-safety categories, not cards. Don't use a harmful-content score as a stand-in for a sensitive information scan. Don't use it as a stand-in for prompt injection detection. For more information about prompt injection protection, see Prompt injection protection for LangChain, LlamaIndex, and Vercel AI SDK apps.

A clean harmful-content score isn't authorization for a later tool. It only means that this text wasn't flagged as harmful. The refund still needs its own gate. For more information about injection, leakage, and unsafe output, see Runtime security for LLM applications. This page stays on the harmful-text hop.

Start with handler-shaped code

A content-moderation check in a request handler is three steps: read the free-text field, call a predicate, and return early on deny. Start with a predicate that you already have, and keep the deny generic. The following handler screens the prompt, calls the provider, and then screens the reply:

1
export async function POST(req: Request) {
2
  const { prompt } = await req.json();
3

4
  if (typeof prompt !== "string" || prompt.length === 0) {
5
    return new Response("Bad request.", { status: 400 });
6
  }
7

8
  if (await isHarmful(prompt)) {
9
    return new Response("Please rephrase your prompt.", { status: 400 });
10
  }
11

12
  const reply = await callProvider({
13
    messages: [{ role: "user", content: prompt }],
14
  });
15

16
  if (await isHarmful(reply)) {
17
    return new Response("I can't show that reply.", { status: 400 });
18
  }
19

20
  return Response.json({ reply });
21
}

The following sections show this hop in four packages. Swap isHarmful for your detector. For more information about the Arcjet rule, see Add one Guard rule.

Add moderation to the Vercel Chat SDK

useChat from @ai-sdk/react sends a POST request with a UIMessage[] body to /api/chat. The screen belongs on that route, not in the hook. This is the Chat SDK path. For more information about the Vercel AI SDK agent guardTool wrap, see How do I secure a Vercel AI SDK agent?.

The following client code sends the thread to /api/chat:

1
import { useChat } from "@ai-sdk/react";
2

3
const { sendMessage } = useChat();
4
// sendMessage({ text: input }) posts the thread to /api/chat.

The following route extracts the most recent user text, decides, and then calls streamText:

1
import { openai } from "@ai-sdk/openai";
2
import type { UIMessage } from "ai";
3
import { convertToModelMessages, isTextUIPart, streamText } from "ai";
4

5
export async function POST(req: Request) {
6
  const { messages }: { messages: UIMessage[] } = await req.json();
7
  const lastMessage = (messages.at(-1)?.parts ?? [])
8
    .filter(isTextUIPart)
9
    .map((p) => p.text)
10
    .join(" ");
11

12
  if (await isHarmful(lastMessage)) {
13
    return new Response("Please rephrase your message.", { status: 400 });
14
  }
15

16
  const result = streamText({
17
    model: openai("gpt-4o"),
18
    messages: await convertToModelMessages(messages),
19
  });
20
  return result.toUIMessageStreamResponse();
21
}

streamText returns a result object rather than a promise, so don't await it. convertToModelMessages does return a promise. Streaming to the browser is a display hop. If you need to screen the completion, then buffer the full string first. Don't treat tokens that already reached the client as unsent.

Add moderation to LangChain in Python

Run the check on the route, then call ainvoke on the model only after an allow. This is a chain hop. For more information about the LangChain Python agent guard_tool wrap, see How do I secure a LangChain Python agent?.

The following FastAPI route screens the prompt, invokes the model, and then screens the reply:

1
from fastapi import FastAPI, HTTPException
2
from langchain.chat_models import init_chat_model
3
from pydantic import BaseModel
4

5
app = FastAPI()
6
llm = init_chat_model("openai:gpt-4o")
7

8

9
class GenerateBody(BaseModel):
10
    prompt: str
11

12

13
@app.post("/v1/generate")
14
async def generate(body: GenerateBody) -> dict[str, str]:
15
    if await is_harmful(body.prompt):
16
        raise HTTPException(
17
            status_code=400,
18
            detail="Please rephrase your prompt.",
19
        )
20

21
    result = await llm.ainvoke(body.prompt)
22
    content = result.content
23
    text = content if isinstance(content, str) else str(content)
24

25
    if await is_harmful(text):
26
        raise HTTPException(status_code=400, detail="I can't show that reply.")
27

28
    return {"reply": text}

Call ainvoke from async code and invoke from sync code. Scan body.prompt, not the whole request model.

Add moderation to the Claude SDK

Screen the prompt, then call messages.create. If the check denies the prompt, then skip the Anthropic call. The following function screens the prompt, calls the Messages API, and then screens the reply:

1
import Anthropic from "@anthropic-ai/sdk";
2

3
const anthropic = new Anthropic();
4

5
export async function generateReply(prompt: string) {
6
  if (await isHarmful(prompt)) {
7
    throw new Error("Please rephrase your prompt.");
8
  }
9

10
  const message = await anthropic.messages.create({
11
    model: "claude-sonnet-4-5",
12
    max_tokens: 1024,
13
    messages: [{ role: "user", content: prompt }],
14
  });
15

16
  const reply = message.content
17
    .filter((block) => block.type === "text")
18
    .map((block) => block.text)
19
    .join("\n");
20

21
  if (await isHarmful(reply)) {
22
    throw new Error("I can't show that reply.");
23
  }
24

25
  return reply;
26
}

This is the Messages API. It isn't the Claude Agent SDK hook path.

Add moderation to the OpenAI SDK

The OpenAI SDK takes the same check in two places: before a chat completion, and before an image generation. The prompt is text in both cases. The following functions screen the prompt, then call the OpenAI SDK:

1
import OpenAI from "openai";
2

3
const openai = new OpenAI();
4

5
export async function generateReply(prompt: string) {
6
  if (await isHarmful(prompt)) {
7
    throw new Error("Please rephrase your prompt.");
8
  }
9

10
  const completion = await openai.chat.completions.create({
11
    model: "gpt-4o",
12
    messages: [{ role: "user", content: prompt }],
13
  });
14

15
  const reply = completion.choices[0]?.message.content ?? "";
16
  if (await isHarmful(reply)) {
17
    throw new Error("I can't show that reply.");
18
  }
19

20
  return reply;
21
}
22

23
export async function generateImage(prompt: string) {
24
  if (await isHarmful(prompt)) {
25
    throw new Error("Please rephrase your prompt.");
26
  }
27

28
  return openai.images.generate({
29
    model: "gpt-image-1",
30
    prompt,
31
  });
32
}

Don't treat the OpenAI Moderation API as a PII scan. If you call it, then it's another harmful-content score on the same string, and you still have to fail the hop yourself.

Add one Guard rule

The following example uses Arcjet content moderation. Content moderation is a Guard rule. There's no protect() variant. Configure the rule once. Bind the untrusted text at the action.

The documented APIs are moderateContent() in JavaScript, ModerateContent() in Python, and GuardModerateContent in Go. Go requires Mode (ModeLive or ModeDryRun). The decision is ALLOW or DENY, the reason is MODERATE_CONTENT, and the per-rule result is a binary detected flag. Optional billing is in text_units. The per-rule result doesn't include category scores. Usage is priced at $0.20 per 1K text units.

Earlier releases exported these rules under an experimental_ prefix. experimental_moderateContent in JavaScript and experimental_ModerateContent in Python remain as deprecated aliases of the same rule.

The following example uses JavaScript or TypeScript:

1
import { launchArcjet, moderateContent } from "@arcjet/guard";
2

3
const arcjet = launchArcjet({ key: process.env.ARCJET_KEY! });
4
const moderate = moderateContent();
5

6
export async function POST(req: Request) {
7
  const { prompt } = await req.json();
8

9
  if (typeof prompt !== "string" || prompt.length === 0) {
10
    return new Response("Bad request.", { status: 400 });
11
  }
12

13
  const decision = await arcjet.guard({
14
    label: "generate.inbound",
15
    rules: [moderate(prompt)],
16
  });
17

18
  if (decision.conclusion === "DENY") {
19
    return new Response("Please rephrase your prompt.", { status: 400 });
20
  }
21

22
  const image = await createImage(prompt);
23
  return Response.json({ image });
24
}

moderateContent() takes an optional config and returns a rule that you bind to a string at the action.

The following example uses Python:

1
import os
2

3
from arcjet.guard import ModerateContent, launch_arcjet
4

5
aj = launch_arcjet(key=os.environ["ARCJET_KEY"])
6
moderate = ModerateContent()
7

8

9
async def screen(text: str) -> bool:
10
    decision = await aj.guard(
11
        label="generate.inbound",
12
        rules=[moderate(text)],
13
    )
14
    return decision.conclusion != "DENY"

launch_arcjet builds the async client. Use launch_arcjet_sync in Flask, Django, and other sync code.

How do I roll out the check without blocking good prompts?

Roll out content moderation in dry run mode first. In dry run the detector still runs and the handler still calls the model, but a deny is only recorded instead of being enforced. Look at a sample of your own traffic, then switch the same rule to live.

The Arcjet content moderation documentation calls the log-only switch mode: "DRY_RUN" in JavaScript and Python, and ModeDryRun in Go. The following example configures that switch:

1
import { launchArcjet, moderateContent } from "@arcjet/guard";
2

3
const arcjet = launchArcjet({ key: process.env.ARCJET_KEY! });
4
const moderate = moderateContent({ mode: "DRY_RUN" });
5

6
const decision = await arcjet.guard({
7
  label: "generate.inbound",
8
  rules: [moderate(prompt)],
9
});

mode is optional in JavaScript and Python and defaults to "LIVE". Go rejects an empty Mode. Switch mode to "LIVE" when you're ready to fail the hop. Keep the user-facing body generic on every path: dry run, live deny, and error.