Bot detection techniques for developers

What problems do bots cause?

Bot detection is the practice of classifying automated HTTP clients and enforcing a policy: allow a verified crawler, constrain a script, or deny abuse. Well-behaved bots have always been part of the web. Search indexers follow robots.txt, fetch at a human-like pace, and send traffic back. Bad bots showed up the moment you exposed a public IP: WordPress scanners, leaked .git probes, SSH brute force, SMTP relay attempts.

Industry measurements now put automated traffic at roughly half of the public web, and in some years a majority. Imperva's 2026 Bad Bot Report attributed more than 53% of observed web traffic in 2025 to automation. Cloudflare has published similar majority-bot figures for HTML requests. Treat those percentages as dated snapshots, not a constant. The direction is stable: machines send a large share of requests, and much of that traffic is not playing fair.

The damage is concrete:

• Expensive requests. Bots hit dynamic pages that cannot be cached: commit history, faceted search, personalized storefronts. Open-source hosts have documented crawlers walking every Git blame and history view.
• Large downloads. ISO mirrors, image archives, and documentation sites absorb bandwidth until the bill or the uplink fails. Projects have had to rate-limit or geo-block entire regions during scrape waves.
• Resource exhaustion. Every request costs CPU, memory, or a downstream query. Brute-force login and relay probing create denial-of-service conditions even when no vulnerability is exploited. Serverless pricing makes "just another request" an invoice.

Good bots mimic human pacing and honor robots.txt. You cannot assume bad bots will. Detection starts with cheap signals and adds harder ones where those fail.

Are AI bots worse than traditional crawlers?

Attribution is messy because User-Agents and IPs are easy to spoof. Traffic logs from open infrastructure still show AI crawlers as a major cost center. Public write-ups have attributed large fractions of some sites' hits to GPTBot, ClaudeBot, or Amazonbot. Read the Docs reported that applying an AI-crawler block list cut bandwidth from about 800 GB/day to about 200 GB/day. Other operators have described crawlers that omit a proper User-Agent entirely.

Incentives explain the hostility. Site owners tolerate Googlebot because the crawl is bounded and search referrals pay for it. robots.txt is enough when the operator cooperates. Training crawls often return no citation and no traffic. If the economic trade is one-way, operators hide. That is why User-Agent blocking alone fails: the clients you most want to stop are the ones least likely to tell the truth.

Not every AI client is a trainer. Search-style fetchers that power answers can send referral traffic. Treat each class separately instead of blocking "all AI."

What happens when agents act on behalf of humans?

Most owners want human traffic. "Good versus bad" is really "is this automation acceptable on this route?" API clients and search indexers are usually acceptable. Scrapers and training crawlers often are not. Agents that act for a person sit in the middle: they hold cookies, complete checkout, or summarize a page the user asked about.

OpenAI's published bots illustrate the split. The following table is a maintained snapshot of how those roles differ. Names and identification methods change, so confirm the current OpenAI crawler docs before you write a rule.

Blocking every AI client is a blunt instrument. Allowing a search-style bot can keep you visible as users move from classic search to answer engines. Blocking a training crawler can save bandwidth. Denying a browser agent on /signup is a different decision from allowing it on /docs. robots.txt is the polite version of that split. Enforce it in the application, because the clients that ignore the file are the ones you care about.

How do you detect and block bots?

Write robots.txt first. It forces you to decide what you want. Google and other well-behaved crawlers follow it. Then assume the rest will not, and stack defenses from cheap to expensive: headers, IP reputation and verification, TLS or HTTP fingerprints, rate limits, then challenges.

A surprising number of abusive clients still send default library User-Agents (curl, python-urllib, Go-http-client). Libraries such as isbot (Node.js) or CrawlerDetect (PHP and Python) catch those. Two limits remain. New bots appear constantly, and any client can copy Googlebot's header.

Arcjet's detectBot rule uses an allow or deny list of named bots and managed categories. Pass exactly one of allow or deny. An empty allow list blocks every detected bot. Categories such as CATEGORY:SEARCH_ENGINE stay current as the underlying list updates.

1
import arcjet, { detectBot, shield, fixedWindow } from "@arcjet/next";
2

3
const aj = arcjet({
4
  key: process.env.ARCJET_KEY!,
5
  rules: [
6
    shield({ mode: "LIVE" }),
7
    detectBot({
8
      mode: "LIVE",
9
      allow: ["CATEGORY:SEARCH_ENGINE"],
10
    }),
11
    fixedWindow({ mode: "LIVE", window: "1h", max: 100 }),
12
  ],
13
});
14

15
export async function GET(req: Request) {
16
  const decision = await aj.protect(req);
17
  if (decision.isDenied()) {
18
    if (decision.reason.isBot()) {
19
      return new Response("Forbidden", { status: 403 });
20
    }
21
    if (decision.reason.isRateLimit()) {
22
      return new Response("Too Many Requests", { status: 429 });
23
    }
24
    return new Response("Forbidden", { status: 403 });
25
  }
26
  return new Response("OK");
27
}

Pin @arcjet/next to v1. After an allow decision, still verify the bot. A client can present a search-engine User-Agent from the wrong network.

The rule is the same shape outside JavaScript. In Python the categories are an enum and the spoof check is a plain function over the results:

1
from arcjet import BotCategory, Mode, arcjet, detect_bot, is_spoofed_bot, shield
2
from fastapi import FastAPI, Request
3
from fastapi.responses import JSONResponse
4

5
app = FastAPI()
6

7
aj = arcjet(
8
    key=ARCJET_KEY,
9
    rules=[
10
        shield(mode=Mode.LIVE),
11
        detect_bot(mode=Mode.LIVE, allow=[BotCategory.SEARCH_ENGINE]),
12
    ],
13
)
14

15

16
@app.get("/")
17
async def index(request: Request):
18
    decision = await aj.protect(request)
19

20
    if decision.is_denied():
21
        return JSONResponse({"error": "Forbidden"}, status_code=403)
22

23
    # Allowed by name, but the address does not back the claim.
24
    if any(is_spoofed_bot(result) for result in decision.results):
25
        return JSONResponse({"error": "Forbidden"}, status_code=403)
26

27
    return {"message": "ok"}

Use arcjet_sync and drop the await for Flask or Django. Python takes integer seconds for rate-limit windows rather than duration strings, and denial reasons are read as decision.reason_v2.type, not reason.isBot().

How do you verify claimed user agents?

Verification answers "is this really Googlebot?" Large operators publish a method: Applebot, Bing, Datadog, Google, and OpenAI among them. The common pattern is reverse DNS plus a forward confirmation. Some also publish IP lists.

To verify a Google crawler:

1. Reverse-lookup the client IP. Confirm the PTR hostname ends in googlebot.com, google.com, or googleusercontent.com.
2. Forward-lookup that hostname. Confirm the address matches the original IP.

If either step fails, treat the request as spoofed. For more information about impersonation and the isSpoofedBot / isVerifiedBot helpers, see detect bot spoofing.

How useful is IP reputation?

Reputation is a prior, not an identity. If an address recently sent bot traffic, the next request from it is more likely automated. Commercial databases (MaxMind, IPinfo, and others) attach owner, connection type, and risk scores. Hosting and cloud ranges are disproportionately automated, which is why signup forms often challenge or deny them.

Treat single-provider percentages as historical. One widely cited 2024 Cloudflare figure put AWS at 12.7% of observed bot traffic. That number will not stay current. The useful fact is the pattern: a few cloud networks originate a lot of automation, and operators sometimes block entire countries or ASNs during a spike. Those blocks have collateral damage.

IP data is imperfect. Geolocation is wrong for mobile and satellite networks. Attackers rotate addresses and buy residential proxies. Blocking solely on reputation creates false positives. Use the signal to raise friction (review an order, tighten a quota) rather than as a permanent country ban. Arcjet returns IP analysis on the decision so your handler can choose the response.

Do proof-of-work and CAPTCHA still work?

Challenges raise the cost of each request. A human can spend a fraction of a second. A crawler doing millions of fetches cannot, in theory. Proof-of-work reverse proxies (Anubis, Checkpoint, and similar) implement that idea. CAPTCHAs are the interactive version: distorted text, image grids, or invisible behavioral scores.

Both are an arms race. Modern vision models solve many image challenges. Human solver services finish the rest for a fraction of a cent. Proof-of-work hurts accessibility and still gets evaded or poisoned. Use a challenge as a graduated step-up on a high-value action, not as the foundation. For more information about why puzzles fail as a primary control, see CAPTCHA alternatives.

What are HTTP message signatures and privacy tokens?

RFC 9421 HTTP Message Signatures let an automated client sign the request so you can verify the operator without trusting the User-Agent. Cloudflare and some AI platforms have pushed this for bots. It adds non-repudiation. It does not replace reverse-DNS verification for classic crawlers. For more information about when to use signatures versus IP checks, see identify AI agents and bots.

Apple shipped a related browser primitive, Private Access Tokens, in 2022. RFC 9577 (Privacy Pass HTTP Authentication Scheme) was published in June 2024. Adoption outside Apple's ecosystem remains limited. Safari can participate. Other browsers have not shipped comparable built-in support, though extensions exist. Do not design your only bot control around a token that most clients cannot present.

How do JA3 and JA4 fingerprints help?

JA3 hashed TLS ClientHello fields so the same client stack produced the same fingerprint across IPs. Reordering cipher suites was enough to change the hash, so JA3 is largely deprecated. JA4 is the successor. Both require access to the TLS handshake. If you terminate TLS on Vercel, Netlify, or Fly.io, you may only see a vendor-injected header, or nothing.

JA4H hashes HTTP request metadata instead. It is proprietary, unlike JA4. Once you have a hash you still have to decide which hashes to deny, the same problem as IP denylists. Combine fingerprints with reputation and application context. Don't block on a hash alone.

How should you rate limit bots?

Per-IP limits stop naive scripts and fail against rotation. Key the limit on a session, user ID, API key, or a compound of IP, path, and fingerprint. Allow verified search crawlers a different budget from anonymous clients. For more information about algorithm choice, see the rate limiting guide.

How do you layer these techniques?

No single technique is enough. Start with robots.txt for operators who cooperate. Classify User-Agents against a maintained list. Verify the ones you allow. Add IP reputation as a prior. Fingerprint when you terminate TLS. Rate-limit the scarce operation, not "HTTP" in general. Keep CAPTCHA and proof-of-work as optional friction.

Put those checks in the request handler so the rule can see the route and the user. Dry-run new denies, then promote them. Review false positives from shared networks, previews, and uptime monitors. Allow CATEGORY:MONITOR and CATEGORY:PREVIEW if those clients matter to you.