What is bot spoofing and how do you detect it?

What is bot spoofing?

Bot spoofing is when a client claims to be a known, usually trusted, automated agent while actually being someone else. The usual disguise is the User-Agent header: a scraper sends Googlebot's string so a naive allowlist lets it through. The request is still a scraper. The name badge is fake.

Spoofing works because many applications treat the User-Agent as identity. robots.txt is keyed on it. CDN rules are keyed on it. Open-source "is this a bot?" libraries are keyed on it. None of those checks prove the operator. They prove the client can write a string.

Verification is the countermeasure. Operators that want you to allow their crawlers publish IP ranges, reverse-DNS names, or both. You resolve the client address and accept the claim only when it matches. Arcjet runs that check for allowlisted bots and exposes the result through @arcjet/inspect helpers: isSpoofedBot and isVerifiedBot.

Verification applies to bots that publish this data: Google, Bing, OpenAI, Apple, Datadog, and others that document how to confirm their crawlers. It is not limited to a fixed set of four names, and it does not cover every automated client. An unknown script has nothing to spoof and nothing to verify. You handle that client with allowlists, reputation, rate limits, and behavior.

For more information about the rest of the detection stack, see bot detection techniques. For more information about signatures when the client is a browser agent rather than a crawler, see identify AI agents and bots.

Why does User-Agent spoofing work?

The User-Agent is a self-asserted label. HTTP/1.1 says clients should send it. It does not say anyone must tell the truth. curl --user-agent "Mozilla/5.0 ... Googlebot/2.1 ..." is a complete spoof. Headless Chrome can send the same header as a person.

Well-behaved bots identify themselves so you can allow them. Googlebot, Bingbot, and OAI-SearchBot all use documented tokens. That honesty helps you and helps impersonators. If your rule is "allow any request whose User-Agent contains Googlebot," you have published the bypass.

Some browsers have reduced or frozen parts of the User-Agent. The header is still present on almost all legitimate traffic, and it is still the name robots.txt uses. Keep classifying it. Stop treating a match as proof.

A missing User-Agent is a different problem. Most real clients send one. You can deny empty values with a filter or inspect the decision and return 400. That is hygiene, not anti-spoofing.

How does reverse DNS verification work?

Reverse DNS (PTR) maps an IP address to a hostname. Forward DNS maps that hostname back to addresses. Verification is the pair of lookups plus a domain check:

1. Take the client IP from a trustworthy source (platform headers your host sets, not a raw X-Forwarded-For the client invented).
2. Look up the PTR record. Confirm the hostname sits under a domain the claimed operator publishes, such as googlebot.com or search.msn.com.
3. Look up the hostname's A or AAAA records. Confirm one of them equals the original client IP.

If the PTR is missing, sits under a consumer ISP, or fails the forward match, the User-Agent is not evidence. The client may be a spoof, a misconfigured proxy, or a bot the operator has not listed. Deny the privileged path (the search-engine allow) and apply your default bot policy.

Some operators skip PTR and publish prefix lists instead. You then test ip in cidr against a cached feed. Refresh the feed. Prefixes change. A stale snapshot both false-allows and false-denies.

Reverse DNS is not instant. Cache positive and negative results for a short TTL. Do the work off the hottest path or use a platform that already did it. Arcjet performs IP and reverse-DNS checks for allow rules so your handler can read the outcome instead of talking to a resolver on every request.

How do you verify Googlebot?

Google documents this procedure and expects you to use it before you grant Googlebot privileges.

Suppose the client IP is 66.249.66.1 and the User-Agent contains Googlebot.

1
$ host 66.249.66.1
2
1.66.249.66.in-addr.arpa domain name pointer
3
  crawl-66-249-66-1.googlebot.com.
4

5
$ host crawl-66-249-66-1.googlebot.com
6
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The PTR is under googlebot.com. The forward lookup returns the same address. Treat this request as verified Googlebot. Allow it on public content if that is your policy.

Now suppose the User-Agent is identical and the IP is a residential or cloud address whose PTR is 1-2-3-4.example-isp.net or is absent. The forward check cannot match a Google hostname. Treat the request as spoofed. Return 403 on any route you reserved for search engines. Do not "give it the benefit of the doubt" because the header looked official.

Bing, Applebot, and OpenAI's documented crawlers follow the same shape with different allowed domains or prefix files. Read the operator's current verification page. Do not copy a hostname suffix from memory.

Which bots can you verify?

You can verify a bot when the operator publishes a confirmation method and you have a claim to check (usually the User-Agent). The public bot list and well-known-bots catalog identifiers. Verification coverage is a subset: the crawlers whose owners want you to distinguish them from impersonators.

Common examples include Google crawlers, Bing, Applebot, OpenAI's documented fetchers, and monitoring products such as Datadog's. The set grows as operators document ranges or DNS. Do not write application logic that assumes "only these four." Do not assume every identifier on the bot list is verifiable. An unknown or spoofable client still belongs in UNKNOWN_BOT or a category deny.

Browser-capable agents are a separate case. They may send a Chrome User-Agent and a CDN IP. Reverse DNS will not say "OpenAI." Some of those agents sign requests with HTTP message signatures. Use that check when you need to recognize them. It is complementary to crawler verification, not a replacement.

How do you detect spoofed bots with Arcjet?

Configure detectBot with an allow list of the bots you want. Arcjet classifies the User-Agent, then verifies allowlisted claims with IP data and reverse DNS. Inspect the results with isSpoofedBot and isVerifiedBot from @arcjet/inspect.

A bot reason also carries isSpoofed() and isVerified(), so decision.reason.isBot() && decision.reason.isSpoofed() works. Prefer the @arcjet/inspect helpers anyway. They run per rule result and skip any rule in DRY_RUN, so a rule you are still evaluating cannot make a request look spoofed to your enforcement branch.

Pin @arcjet/next and @arcjet/inspect to v1.

1
import arcjet, { detectBot, shield, fixedWindow } from "@arcjet/next";
2
import { isSpoofedBot, isVerifiedBot } from "@arcjet/inspect";
3

4
const aj = arcjet({
5
  key: process.env.ARCJET_KEY!,
6
  rules: [
7
    shield({ mode: "LIVE" }),
8
    detectBot({
9
      mode: "LIVE",
10
      allow: ["CATEGORY:SEARCH_ENGINE"],
11
    }),
12
    fixedWindow({ mode: "LIVE", window: "1h", max: 100 }),
13
  ],
14
});
15

16
export async function GET(req: Request) {
17
  const decision = await aj.protect(req);
18

19
  if (decision.results.some(isSpoofedBot)) {
20
    return new Response("Forbidden", { status: 403 });
21
  }
22

23
  if (decision.isDenied()) {
24
    if (decision.reason.isBot()) {
25
      return new Response("Forbidden", { status: 403 });
26
    }
27
    if (decision.reason.isRateLimit()) {
28
      return new Response("Too Many Requests", { status: 429 });
29
    }
30
    return new Response("Forbidden", { status: 403 });
31
  }
32

33
  // Only after the conclusion: a verified crawler is still subject to the
34
  // fixedWindow above, so this branch must not run before isDenied().
35
  if (decision.results.some(isVerifiedBot)) {
36
    return new Response("Hello, verified crawler");
37
  }
38

39
  return new Response("OK");
40
}

An empty allow list denies every detected bot, which is correct for signup and login. On public pages, CATEGORY:SEARCH_ENGINE is the usual allow. Pass exactly one of allow or deny. After an allow decision, still check isSpoofedBot. Classification and verification are different facts.

Start in DRY_RUN if you are unsure who claims to be a search engine on your site. The dashboard shows the claimed identity and whether verification passed.

What should you do with a spoofed or verified bot?

A spoofed bot is not a search engine. Deny it on every route where you would have allowed the real crawler. A 403 is appropriate. Do not issue a CAPTCHA. There is no person to solve it, and a solver farm will.

A verified bot is a known operator, not a trusted user. Allow it only where that operator has a job: public HTML, sitemaps, product listings you want indexed. Deny it on /login, /signup, /checkout, password reset, and private APIs. Verification answers "who sent this?" Authorization answers "may they do this?"

Unknown automation still exists. A strict allowlist blocks it. A permissive deny list lets unidentified clients through. That is a policy choice. Write it per route, measure it, and keep the spoof check on any path that grants crawler privileges.

See the bot protection reference for decision fields and error handling.