How to identify AI agents and bots

What is AI agent identification?

AI agent identification is the practice of determining whether an HTTP client is a named automated agent, then verifying that claim, so you can allow useful crawlers and assistants while denying impersonators. The hard part is not "blocking bots." It is distinguishing a malicious scraper from a search indexer, a training crawler, or a browser-capable assistant that a customer sent.

You already decide what automated traffic you want. Should your catalog appear in AI answers? Should an agent sign up or check out on a user's behalf? Do you want Googlebot on every route? Identification is how you enforce that policy without trusting a string the client wrote about itself.

Three methods dominate today: the User-Agent header, IP and reverse-DNS verification, and HTTP message signatures. Use them together. A header tells you who the client claims to be. Verification tells you whether that claim is true. Signatures help when the agent uses a real browser and no longer looks like a crawler.

For more information about layered detection, see bot detection techniques. For more information about impersonation, see detect bot spoofing.

Comparison of identification methods

The following table scores the three methods on reliability, spoofability, and setup cost. Scores are relative, not absolute. A high-reliability method still fails if you skip key rotation or treat an outdated IP prefix as current.

Treat published prefixes, key IDs, and browser version strings as maintained data. Operators rotate them. Examples later in this article are illustrative, not a list to hard-code forever.

How reliable is the User-Agent header?

The User-Agent header is the lowest-effort identification signal and the weakest. HTTP clients are supposed to send it. Nothing stops them from sending anything they like.

A terminal client identifies itself by default. curl sends a string such as curl/8.7.1. A browser sends a long Mozilla-compatible string. Well-behaved bots identify themselves truthfully. OpenAI's search indexer uses a dedicated string such as OAI-SearchBot/1.0; +https://openai.com/searchbot. Googlebot uses a documented Googlebot token inside a browser-like string. Arcjet tracks hundreds of these identifiers in the open source well-known-bots project and the public bot list.

That honesty is voluntary. You can change curl's header with --user-agent "hello" and the server sees hello. Malicious scrapers set a current Chrome string and skip past naive allowlists. Chrome ships a new stable milestone every few weeks, so any allowlist that hard-codes a version number is stale within the month; at the time of writing stable is on Chrome 151, which is thirteen milestones past where a list written in mid-2025 would have stopped. Use the header to classify a claim, then verify the claim.

An empty or missing User-Agent is itself a signal. Most legitimate clients send the header because HTTP/1.1 says they should. You can deny those requests with a filter, or inspect the decision and return 400.

How do you verify a bot by IP and reverse DNS?

IP verification checks that the source address belongs to the operator named in the User-Agent. Large crawler operators publish this data so you can allow their indexer without allowing every client that copies the header.

Reverse DNS is the usual implementation, because it avoids shipping a constantly changing prefix list in your app:

1. Resolve the PTR record for the client IP. Confirm the hostname sits under an expected domain, such as googlebot.com, google.com, or search.msn.com.
2. Resolve that hostname forward (A or AAAA). Confirm the result matches the original client IP.

If either step fails, the client is spoofing or the operator has not published verification data for that bot. Do not allow it as a search engine.

Some operators also publish machine-readable IP prefixes. OpenAI documents SearchBot prefixes as JSON that you can fetch and cache. A containment check against a CIDR is then enough. Prefixes change, so treat any example range as illustrative and refresh the list. Hard-coding a snapshot such as a /28 from a blog post allows the wrong network or blocks the real crawler once the operator renumbers.

IP verification lets you trust the User-Agent for named crawlers. You can allow verified Google and OpenAI indexers on public content and deny everyone else who claims those names. It does not identify a browser-capable agent. That client looks like Chrome, often arrives from a CDN or hosting range, and may geolocate near the human who launched it.

How do HTTP message signatures identify browser agents?

Browser-capable agents break the crawler model. ChatGPT's agent mode, which launched in mid-2025, is one example: the product opens a real browser, follows instructions, and sends a standard Chrome User-Agent. In one captured session the source IP belonged to Cloudflare, not OpenAI, and geolocated near the person talking to ChatGPT. User-Agent and IP both said "ordinary browser on a hosting network."

Some platforms now sign outbound HTTP with RFC 9421 HTTP Message Signatures and a Signature-Agent header. ChatGPT's agent has used Signature-Agent: "https://chatgpt.com" (quotes included) and advertised keys from a /.well-known/http-message-signatures-directory document. Verification then looks like this:

1. Confirm Signature-Agent matches the expected origin, including quoting.
2. Fetch the current key set from that origin's well-known directory.
3. Verify Signature against Signature-Input with the key ID, algorithm, created and expiry timestamps, and covered components.

The following TypeScript sketch uses web-bot-auth. The key material and header values are illustrative. Fetch live keys. Do not paste expired kid or x values from an old capture into production.

1
import { verify } from "web-bot-auth";
2
import { verifierFromJWK } from "web-bot-auth/crypto";
3

4
// Replace with a JWK from the operator's current well-known directory.
5
const jwk = await fetchCurrentJwk(keyidFromSignatureInput);
6

7
const signedRequest = new Request("https://example.com/", {
8
  headers: {
9
    Signature: request.headers.get("Signature") ?? "",
10
    "Signature-Input": request.headers.get("Signature-Input") ?? "",
11
    "Signature-Agent": request.headers.get("Signature-Agent") ?? "",
12
  },
13
});
14

15
await verify(signedRequest, await verifierFromJWK(jwk));

Current Node.js releases can run TypeScript files directly, so you can prototype this without a separate compile step. In production, cache the key set, enforce expiry, and fail closed when the signature is missing on a route that requires a named agent.

Signatures do not replace IP verification for crawlers. They cover the case IP verification cannot: a real browser, a CDN egress IP, and a platform that is willing to non-repudiate the request.

How do you combine methods in application code?

Start with a known-bot allowlist, then inspect verification. Arcjet's detectBot rule classifies the User-Agent against the bot list and, for allow rules, verifies claimed bots with IP data and reverse DNS. You then branch on isDenied(), spoofed results, and verified results.

The following example allows search-engine crawlers, denies other detected bots, and uses @arcjet/inspect helpers. Pin @arcjet/next and @arcjet/inspect to v1.

1
import arcjet, { detectBot, shield, fixedWindow } from "@arcjet/next";
2
import { isSpoofedBot, isVerifiedBot } from "@arcjet/inspect";
3

4
const aj = arcjet({
5
  key: process.env.ARCJET_KEY!,
6
  rules: [
7
    shield({ mode: "LIVE" }),
8
    detectBot({
9
      mode: "LIVE",
10
      allow: ["CATEGORY:SEARCH_ENGINE"],
11
    }),
12
    fixedWindow({ mode: "LIVE", window: "1h", max: 100 }),
13
  ],
14
});
15

16
export async function GET(req: Request) {
17
  const decision = await aj.protect(req);
18

19
  if (decision.results.some(isSpoofedBot)) {
20
    return new Response("Forbidden", { status: 403 });
21
  }
22

23
  // Check the conclusion before the verified-bot branch. Returning early on
24
  // a verified crawler would hand it an unlimited budget, because the
25
  // fixedWindow above has not been acted on yet.
26
  if (decision.isDenied()) {
27
    if (decision.reason.isBot()) {
28
      return new Response("Forbidden", { status: 403 });
29
    }
30
    if (decision.reason.isRateLimit()) {
31
      return new Response("Too Many Requests", { status: 429 });
32
    }
33
    return new Response("Forbidden", { status: 403 });
34
  }
35

36
  if (decision.results.some(isVerifiedBot)) {
37
    return new Response("Hello, verified crawler");
38
  }
39

40
  return new Response("OK");
41
}

Order matters here. isVerifiedBot is an identity fact, not an allowance, so branching on it before isDenied() would return 200 to a verified crawler that had already blown through the fixedWindow in the same config. Check the conclusion first, then use verification to decide what an allowed client gets.

An empty allow list denies every detected bot. A named category such as CATEGORY:SEARCH_ENGINE is the usual public-content default. Pass exactly one of allow or deny. Do not use a block list. After an allow decision, still check spoofing. A client can present a Googlebot User-Agent from a residential proxy.

The same three-way branch in Go, where the spoof and verify checks are methods on the decision instead of a separate package. Individual bot names such as OPENAI_CRAWLER_SEARCH can sit alongside a category:

1
aj, err := arcjet.NewClient(arcjet.Config{
2
  Rules: []arcjet.Rule{
3
    arcjet.DetectBot(arcjet.BotOptions{
4
      Mode: arcjet.ModeLive,
5
      Allow: []string{
6
        arcjet.BotCategorySearchEngine,
7
        "OPENAI_CRAWLER_SEARCH",
8
      },
9
    }),
10
  },
11
})
12
if err != nil {
13
  return err
14
}
15

16
http.HandleFunc("GET /", func(w http.ResponseWriter, r *http.Request) {
17
  decision, _ := aj.Protect(r.Context(), r)
18

19
  // Denied: the claimed agent is not on the allow list.
20
  if decision.IsDenied() {
21
    http.Error(w, "Forbidden", http.StatusForbidden)
22
    return
23
  }
24

25
  // Allowed by name, but the address does not back the claim.
26
  if decision.IsSpoofedBot() {
27
    http.Error(w, "Forbidden", http.StatusForbidden)
28
    return
29
  }
30

31
  if decision.IsVerifiedBot() {
32
    w.Header().Set("X-Robots-Tag", "index")
33
  }
34

35
  w.Write([]byte("ok"))
36
})

Protect returns an error separately from the decision, and a transport failure fails open with a usable decision, so check IsDenied() rather than treating a non-nil error as a block. The Go SDK is pre-release; pin an exact version.

HTTP signature verification is a separate check you add when you want to recognize a browser agent that is not on the crawler list. Compose it with the bot decision: allow a verified signature on /docs, and still deny that same client on /login.

What policy should you apply after you know who it is?

Identification is not permission. A verified search crawler is still not authorized to hit a private API. A signed shopping agent is still not allowed to stuff credentials. Write the rule per route.

Useful defaults:

• Allow verified search-engine indexers on public pages. Deny unverified clients that only copy those User-Agents.
• Decide explicitly what to do with CATEGORY:AI on docs versus login.
• Treat hosting and CDN IPs as automated until a signature or another strong signal says otherwise. Don't blanket-block them on an API that is supposed to receive server-side clients.
• Allow a named, verified agent on a confined path (catalog, docs) and deny it on signup, password reset, and checkout unless you have a commerce policy for delegated purchase.

Start those rules in DRY_RUN. Measure how much login and catalog traffic would have been denied. Then promote route by route. For more information about classifying agents as clients rather than blocking every bot, see AI agent bot management.