How to block AI crawlers but allow AI agents

Keep training crawlers out, let agents that act for people in, and require IP or signature evidence before you trust an agent's name.

14 min read
In short: An AI crawler collects pages ahead of time for model training or a search index, and an AI agent fetches a page on demand for a person who is waiting on the answer. To block crawlers but allow agents, allow CATEGORY:AI_AGENT in an Arcjet detectBot rule and leave CATEGORY:AI out, then accept an agent only when its bot identity has IP or SIGNATURE evidence on routes where a forged claim matters. Keep robots.txt entries as the documented training opt-out, because robots.txt states a preference and doesn't enforce it.

How do you block AI crawlers but allow AI agents?

To block AI crawlers but allow AI agents, allow CATEGORY:AI_AGENT in a bot rule and leave CATEGORY:AI out, then require verified evidence for agents on routes where a forged claim would cost you. An AI crawler collects pages ahead of time, for model training or a search index. An AI agent fetches a page on demand for a person who is waiting on the answer.

Most sites want those two groups treated differently. Training crawlers copy your content into a model without sending anyone back to you. Agents act for people who are asking about your product, comparing your prices, or trying to buy from you. Blocking every AI bot turns those people away, and allowing every AI bot gives your catalog to every training run that asks.

The following sections explain how to tell the two groups apart, why robots.txt doesn't enforce the difference, and how to enforce it in your request handler with Arcjet in JavaScript, Python, and Go.

For more information about how bots identify themselves and how you check the claim, see how to identify AI agents and bots.

What's the difference between an AI agent and an AI crawler?

An AI crawler visits pages on its own schedule and stores what it collects, for model training or an AI search index. An AI agent visits a page because a person asked a question or gave it a task, and it uses the page to answer that person. The same company often runs both, under different names, so you can't decide by operator.

OpenAI and Anthropic each document separate bots for training, search, and user requests. OpenAI's bot documentation states that each of its settings is independent of the others, and Anthropic's crawler documentation describes the same three-way split. The following table maps them to Arcjet bot IDs, as of November 2026:

OperatorBotWhat it doesArcjet bot IDKind
OpenAIGPTBotCrawls content that may be used to train OpenAI's modelsOPENAI_CRAWLERAI_CRAWLER
OpenAIOAI-SearchBotIndexes sites for ChatGPT searchOPENAI_CRAWLER_SEARCHAI_CRAWLER
OpenAIChatGPT-UserVisits a page when a ChatGPT user asks about itOPENAI_CRAWLER_USERAI_AGENT
OpenAIChatGPT agentRuns a cloud browser to complete a task, and signs its requestsCHATGPT_AGENTAI_AGENT
AnthropicClaudeBotCollects content that may be used to train Anthropic's modelsANTHROPIC_CRAWLERAI_CRAWLER
AnthropicClaude-SearchBotIndexes content to improve Claude's search resultsANTHROPIC_CRAWLER_SEARCHAI_CRAWLER
AnthropicClaude-UserFetches a page when a Claude user asks a question that needs itANTHROPIC_CRAWLER_USERAI_AGENT

ChatGPT agent doesn't name itself in its User-Agent header. It sends a Chrome User-Agent and signs each request with HTTP Message Signatures, with a Signature-Agent header set to "https://chatgpt.com", according to OpenAI's ChatGPT agent allowlisting article. Arcjet verifies that signature with Web Bot Auth and reports the operator as chatgpt.com.

Arcjet puts every agent in CATEGORY:AI_AGENT, and every bot in that category is also in CATEGORY:AI. The crawlers are the rest of CATEGORY:AI. When Arcjet Cloud detects a bot, it reports a bot identity whose kind is AI_AGENT for an agent and AI_CRAWLER for a crawler, so your code can read the distinction directly.

Search crawlers need a decision of their own. OAI-SearchBot is in both CATEGORY:AI and CATEGORY:SEARCH_ENGINE, so allowing search engines allows it. Claude-SearchBot is only in CATEGORY:AI, so you add ANTHROPIC_CRAWLER_SEARCH to the allow list if you want your pages in Claude's search results.

Why isn't robots.txt enough?

robots.txt is a request that a crawler can honor or ignore, not an access control. It tells honest crawlers what you want, and it doesn't stop a client that ignores it or that claims another bot's name. Use it to state your preference, and enforce the preference in code.

Three gaps follow from that design:

  • Some agents don't apply it. OpenAI's documentation says of ChatGPT-User that "because these actions are initiated by a user, robots.txt rules may not apply." Anthropic states that its bots, including Claude-User, honor robots.txt. Behavior differs by operator, and it can change.
  • Scrapers don't read it. A scraper that copies GPTBot's or ChatGPT-User's User-Agent reads whatever it wants. robots.txt has no way to check who sent a request.
  • It can't see the request. robots.txt applies the same rule to every path matched by a prefix. It can't allow an agent on your pricing page while it denies the same agent on your signup form, and it can't rate limit.

Keep your robots.txt entries anyway. A Disallow for GPTBot or ClaudeBot is the opt-out signal that OpenAI and Anthropic document for model training, and Google uses the Google-Extended product token the same way. Google's crawler documentation states that Google-Extended has no separate user agent string and doesn't affect inclusion in Google Search. A server-side rule enforces what robots.txt asks for. It doesn't replace the signal that the operators read.

How do you allow AI agents and block AI crawlers with Arcjet?

Add a detectBot rule with an allow list that includes CATEGORY:AI_AGENT and leaves out CATEGORY:AI. A rule that denies CATEGORY:AI also blocks AI agents, because every agent is in that category too.

An allow list blocks every other detected bot, so include the other categories that your site relies on, such as CATEGORY:SEARCH_ENGINE. The rule runs in your request handler, so you can attach it to the routes where it applies and use a different rule elsewhere.

Allow AI agents in JavaScript

The following rule allows search engines and AI agents, and denies AI crawlers and every other detected bot:

detectBot({
mode: "LIVE", // Blocks requests. Use "DRY_RUN" to log only
// Block all bots except search engines and AI agents
allow: [
"CATEGORY:SEARCH_ENGINE", // Google, Bing, etc
"CATEGORY:AI_AGENT", // ChatGPT-User, Claude-User, coding agents, etc
],
}),

Allow AI agents in Python

The Python SDK takes the same categories through the BotCategory enum:

from arcjet import BotCategory, detect_bot, Mode
detect_bot(
mode=Mode.LIVE, # Blocks requests. Use Mode.DRY_RUN to log only
# Block all bots except search engines and AI agents
allow=[
BotCategory.SEARCH_ENGINE, # Google, Bing, etc
BotCategory.AI_AGENT, # ChatGPT-User, Claude-User, coding agents, etc
],
)

Allow AI agents in Go

The Go SDK takes the same categories as BotCategory* constants:

arcjet.DetectBot(arcjet.BotOptions{
Mode: arcjet.ModeLive, // Blocks requests. Use ModeDryRun to log only.
// Block all bots except search engines and AI agents
Allow: []string{
arcjet.BotCategorySearchEngine, // Google, Bing, etc
arcjet.BotCategoryAIAgent, // ChatGPT-User, Claude-User, coding agents, etc
},
})

To also allow Claude's search crawler, add ANTHROPIC_CRAWLER_SEARCH to the list. To block one agent, such as an agent you don't want on any route, use a separate rule with its bot ID in a deny list.

For more information about the categories and bot IDs, see Allow AI agents and block AI crawlers in the Arcjet docs.

How do you require verified evidence for AI agents?

To stop a scraper that claims to be an AI agent from passing as one, accept an agent only when its bot identity has IP or SIGNATURE evidence. An allow list matches names, and a name in a User-Agent header costs nothing to forge. The evidence level tells you whether Arcjet confirmed the name.

Arcjet checks an agent's claim in the ways its operator supports:

  • IP: The request came from the operator's published address ranges, or reverse DNS confirmed it. OpenAI publishes ranges for ChatGPT-User, and Anthropic publishes them for its bots.
  • SIGNATURE: A Web Bot Auth signature or a Personal Agent Protocol session verified against keys that the operator publishes on its own domain. ChatGPT agent and Meta's Muse are examples.
  • UNVERIFIED: Arcjet matched the User-Agent header and didn't verify it.

When a request claims to be ChatGPT-User from an address outside OpenAI's ranges, Arcjet sets spoofed on the identity. A spoofed identity always has UNVERIFIED evidence, and the isAiAgent helper returns false for it. A claim that Arcjet can't check, such as an address it has no data for, stays UNVERIFIED without spoofed. That second case is the one the evidence check closes.

Coding agents such as Claude Code fetch pages from the developer's own computer, so Arcjet reports them with UNVERIFIED evidence and doesn't mark them as spoofed. If developers read your docs through coding agents, don't require verified evidence on your docs routes. Require it on routes where the agent gets more than a person browsing anonymously would.

Require verified evidence in JavaScript

The following Node.js server allows search engines and AI agents, denies spoofed claims, and accepts an AI agent only with IP or SIGNATURE evidence. isAiAgent comes from @arcjet/inspect:

import arcjet, {
detectBot,
type ArcjetBotIdentity,
type ArcjetDecision,
} from "@arcjet/node";
import { isAiAgent, isSpoofedBot } from "@arcjet/inspect";
import http from "node:http";
const aj = arcjet({
key: process.env.ARCJET_KEY!,
rules: [
detectBot({
mode: "LIVE",
allow: ["CATEGORY:SEARCH_ENGINE", "CATEGORY:AI_AGENT"],
}),
],
});
// Evidence that proves more than a User-Agent claim
const VERIFIED_EVIDENCE = new Set(["IP", "SIGNATURE"]);
// Return the AI agent's identity when Arcjet verified it
function verifiedAiAgent(
decision: ArcjetDecision,
): ArcjetBotIdentity | undefined {
for (const result of decision.results) {
if (isAiAgent(result) && result.reason.isBot()) {
const identity = result.reason.identity;
if (identity && VERIFIED_EVIDENCE.has(identity.evidence)) {
return identity;
}
}
}
return undefined;
}
const server = http.createServer(async (req, res) => {
const decision = await aj.protect(req);
// AI crawlers, other bots, and forged claims
if (decision.isDenied() || decision.results.some(isSpoofedBot)) {
res.writeHead(403, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Forbidden" }));
return;
}
const agent = verifiedAiAgent(decision);
// An AI agent that Arcjet couldn't verify is only a claim
if (decision.results.some(isAiAgent) && !agent) {
res.writeHead(403, { "Content-Type": "application/json" });
res.end(JSON.stringify({ error: "Unverified agent" }));
return;
}
if (agent) {
// For example: CHATGPT_AGENT SIGNATURE chatgpt.com
console.log(agent.botId, agent.evidence, agent.operator);
}
res.writeHead(200, { "Content-Type": "text/html" });
res.end("<h1>Pricing</h1>");
});
server.listen(3000);

Require verified evidence in Python

The following FastAPI route applies the same policy with is_ai_agent and is_spoofed_bot. In Python, the identity is on result.reason_v2.identity, and the bot ID field is bot_id:

import os
from arcjet import (
BotCategory,
BotIdentity,
Decision,
Mode,
arcjet,
detect_bot,
is_ai_agent,
is_spoofed_bot,
)
from fastapi import FastAPI, Request
from fastapi.responses import HTMLResponse, JSONResponse
app = FastAPI()
aj = arcjet(
key=os.environ["ARCJET_KEY"],
rules=[
detect_bot(
mode=Mode.LIVE,
allow=[BotCategory.SEARCH_ENGINE, BotCategory.AI_AGENT],
),
],
)
# Evidence that proves more than a User-Agent claim
VERIFIED_EVIDENCE = {"IP", "SIGNATURE"}
def verified_ai_agent(decision: Decision) -> BotIdentity | None:
"""Return the AI agent's identity when Arcjet verified it."""
for result in decision.results:
reason = result.reason_v2
if is_ai_agent(result) and reason.type == "BOT":
identity = reason.identity
if identity is not None and identity.evidence in VERIFIED_EVIDENCE:
return identity
return None
@app.get("/pricing")
async def pricing(request: Request):
decision = await aj.protect(request)
# AI crawlers, other bots, and forged claims
if decision.is_denied() or any(is_spoofed_bot(r) for r in decision.results):
return JSONResponse({"error": "Forbidden"}, status_code=403)
agent = verified_ai_agent(decision)
# An AI agent that Arcjet couldn't verify is only a claim
if any(is_ai_agent(r) for r in decision.results) and agent is None:
return JSONResponse({"error": "Unverified agent"}, status_code=403)
if agent is not None:
# For example: OPENAI_CRAWLER_USER IP
print(agent.bot_id, agent.evidence, agent.operator)
return HTMLResponse("<h1>Pricing</h1>")

Require verified evidence in Go

The Go SDK reports the identity on result.Reason.Bot.Identity, and decision.IsAIAgent() returns true for a non-spoofed AI agent in a LIVE rule:

package main
import (
"log"
"net/http"
"os"
"github.com/arcjet/arcjet-go"
)
// verifiedAIAgent returns the AI agent's identity when Arcjet verified it.
func verifiedAIAgent(decision arcjet.Decision) *arcjet.BotIdentity {
if !decision.IsAIAgent() {
return nil
}
for _, result := range decision.Results {
bot := result.Reason.Bot
if bot == nil || bot.Identity == nil {
continue
}
id := bot.Identity
if id.Kind != arcjet.BotKindAIAgent || id.Spoofed {
continue
}
if id.Evidence == arcjet.BotEvidenceIP ||
id.Evidence == arcjet.BotEvidenceSignature {
return id
}
}
return nil
}
func main() {
aj, err := arcjet.NewClient(arcjet.Config{
Key: os.Getenv("ARCJET_KEY"),
Rules: []arcjet.Rule{
arcjet.DetectBot(arcjet.BotOptions{
Mode: arcjet.ModeLive,
Allow: []string{
arcjet.BotCategorySearchEngine,
arcjet.BotCategoryAIAgent,
},
}),
},
})
if err != nil {
log.Fatal(err)
}
http.HandleFunc("/pricing", func(w http.ResponseWriter, r *http.Request) {
decision, err := aj.Protect(r.Context(), r)
if err != nil {
http.Error(w, "Internal Server Error", http.StatusInternalServerError)
return
}
// AI crawlers, other bots, and forged claims
if decision.IsDenied() || decision.IsSpoofedBot() {
http.Error(w, "Forbidden", http.StatusForbidden)
return
}
agent := verifiedAIAgent(decision)
// An AI agent that Arcjet couldn't verify is only a claim
if decision.IsAIAgent() && agent == nil {
http.Error(w, "Unverified agent", http.StatusForbidden)
return
}
if agent != nil {
// For example: CHATGPT_AGENT SIGNATURE chatgpt.com
log.Println(agent.BotID, agent.Evidence, agent.Operator)
}
w.Header().Set("Content-Type", "text/html")
w.Write([]byte("<h1>Pricing</h1>"))
})
log.Fatal(http.ListenAndServe(":3000", nil))
}

Only Arcjet Cloud computes the bot identity. A decision that the SDK makes locally, such as a denial from its local bot detection, has no identity, and the AI agent helpers return false for it.

For more information about what each evidence level proves and how an attacker defeats it, see AI agent evidence levels. For more information about forged bot claims, see detect bot spoofing.

What about AI agents that use a normal browser?

Agents that run in a person's own browser, such as Claude in Chrome and Perplexity Comet, send that browser's requests, with the person's cookies, IP address, and User-Agent header. No server-side category, IP check, or signature can tell them apart from the person, so a rule that allows CATEGORY:AI_AGENT doesn't apply to them and a rule that denies CATEGORY:AI doesn't block them.

Cloud browsers are a different case. ChatGPT agent, Manus, and Browserbase sessions also send a browser User-Agent header, but they sign their requests, so Arcjet names them with SIGNATURE evidence. A cloud browser that doesn't sign, such as Instinct, looks like an ordinary browser on a cloud network.

For agents that you can't identify, the following controls still apply:

  • Advanced signals run checks in the page and detect a browser that a program drives. A failed check gives the request an identity with kind AUTOMATION and FINGERPRINT evidence. It tells you the browser is automated, not which agent drives it. For more information, see advanced signals in the Arcjet docs.
  • Rate limits cap actions that people don't repeat at machine speed, such as form posts, searches, and add-to-cart requests.
  • Your own authorization decides what a signed-in account may do. A person who lets an agent drive their browser is accountable for what it does with their session.

For more information about this case, see how to detect AI agents in a person's browser.

What should a blocked crawler receive?

Return 403 Forbidden to an AI crawler that your policy denies. The status tells an honest operator that the denial is deliberate and that retrying won't help, and it costs you less to serve than the page. Reserve 429 Too Many Requests for clients that you allow but that exceed a rate limit.

The following responses fit the other cases:

  • A spoofed claim gets 403. It isn't the bot it claims to be, and it gets nothing that the real bot would get.
  • An AI agent over its budget gets 429, with a Retry-After header if your framework sets one. Give each agent its own budget, keyed by its bot ID, so one agent can't use up capacity that your other visitors need.
  • An AI agent on an account route gets 401 and a pointer to your API or sign-in page. A bot identity doesn't authorize purchases, password changes, or data deletion.

Don't serve a CAPTCHA to a denied crawler. There's no person to solve it. Don't serve the crawler different content from what people see on an indexed page, either: if you want a page out of an AI search index, deny the search crawler or use the operator's opt-out.

How do you roll out the rule without blocking customers?

Start the rule in DRY_RUN mode, measure which AI bots reach each route and with what evidence, and then switch to LIVE route by route. A dry run reports what the rule would have done without changing any response.

Follow these steps:

  1. Set mode to DRY_RUN on the detectBot rule and deploy it.
  2. In the Arcjet dashboard, review which bots the rule would deny on each route.
  3. Add any categories or bot IDs that your site needs, such as ANTHROPIC_CRAWLER_SEARCH or a monitoring service.
  4. Switch the rule to LIVE on public content routes first, then add the evidence check on the routes where an unverified agent shouldn't get through.

The AI agent helpers ignore DRY_RUN results, so a rule that you're still measuring can't change what your code does for an agent.

To start, follow the Arcjet bot protection quick start, then add CATEGORY:AI_AGENT to your allow list. For more information about the agent helpers and per-agent responses, see Detect AI agents and block AI crawlers in the Arcjet docs.

Frequently asked questions

Is ChatGPT-User a crawler or an agent?

An agent. OpenAI documents ChatGPT-User as the fetcher for user-initiated actions in ChatGPT, not for automatic crawling. Arcjet reports it as OPENAI_CRAWLER_USER in CATEGORY:AI_AGENT, with IP evidence when the request comes from OpenAI's published ranges.

Does blocking GPTBot affect ChatGPT search?

No, according to OpenAI's bot documentation, as of November 2026. GPTBot crawls for model training, OAI-SearchBot indexes for ChatGPT search, and OpenAI states that each setting is independent of the others. In Arcjet, OAI-SearchBot is in CATEGORY:SEARCH_ENGINE, so allowing search engines lets it through while GPTBot stays blocked.

Can a crawler pretend to be an agent?

Yes. Any client can send ChatGPT-User in its User-Agent header. Arcjet marks the identity as spoofed when the address contradicts the claim, and reports UNVERIFIED evidence when it can't check it. Require IP or SIGNATURE evidence on routes where a forged agent would cost you.

Should I return 403 or 429 to a blocked AI crawler?

Return 403 to a crawler your policy denies, because retrying won't change the answer. Use 429 for clients you allow that exceed a rate limit, such as an AI agent over its per-agent budget.

Does this affect Googlebot?

No. Googlebot is in CATEGORY:SEARCH_ENGINE, so an allow list that includes search engines keeps it. Google's training opt-out is the Google-Extended robots.txt token, which has no separate user agent string and, according to Google, doesn't affect inclusion in Google Search.

Do I still need robots.txt?

Yes. A Disallow for GPTBot or ClaudeBot is the training opt-out signal that OpenAI and Anthropic document. robots.txt states your preference to honest crawlers, and a rule in your request handler enforces it against clients that ignore it.

Can I block AI agents that run in a person's browser?

Not by category. Agents such as Claude in Chrome and Perplexity Comet send the person's own browser requests. Advanced signals can detect automation in the page, and rate limits and your own authorization still apply.

Application security in your code

Protect your application with Arcjet

Allow CATEGORY:AI_AGENT in the request handler and check the bot identity, so a scraper that copies an agent's name doesn't get what the agent gets.