How do I functionally test security rules?

How do I functionally test security rules?

Send the traffic that should trip the rule to the same app binary you run in production, then assert the status code. Arcjet evaluates protect() locally (WebAssembly) and, when the rule needs shared state, through the decide API. That path is identical on a laptop, in CI, and on Fly. You do not stand up a separate WAF fixture.

Untested rules fail in two directions. They fail open: a bot rule that never matches curl because the handler reads req.headers.get("user-agent") after a rewrite. They fail closed: a rate limit you flip to LIVE in production that returns an HTML error page to an API client, or a window so tight that a legitimate form POST is 429. Both are cheaper to see in CI than in a support thread.

Create the client once with a v1 key and rules array. Put the test target in LIVE mode. DRY_RUN logs the decision but still returns allow, so a functional test against DRY_RUN cannot see a 403 or 429. Use DRY_RUN on the production deploy until you have a day of dashboard data, then switch that rule to LIVE and keep the functional suite on a dedicated test key.

1
import arcjet, { detectBot, fixedWindow, shield } from "@arcjet/node";
2

3
const aj = arcjet({
4
  key: process.env.ARCJET_KEY!,
5
  rules: [
6
    shield({ mode: "LIVE" }),
7
    detectBot({ mode: "LIVE", allow: [] }),
8
    fixedWindow({ mode: "LIVE", window: "3s", max: 50 }),
9
  ],
10
});

Route-handler unit tests (mocked protect(), mocked auth()) live in testing Next.js App Router API routes. This page is the complementary suite: real HTTP against a running server. Use OrbStack HTTPS if production is TLS-only and cookies or HSTS depend on it.

How do I test each Arcjet rule type?

Shield's x-arcjet-suspicious header is a documented test flag. Do not send attack payloads at production to "exercise the WAF." Rate limits need a short window in the test environment (3 seconds in the example) so the suite finishes. Bot tests should assert the JSON body your handler returns, not an HTML error page from a generic gateway.

Which load tool should I use?

Newman, k6, and Artillery are all current. They are not interchangeable.

Prefer Newman or a one-VU k6 script for security rules. A 200-VU k6 run against a 50-request window will trip the limit and tell you nothing about which request should have been the first deny.

How do I run a Newman collection against a rate limit?

Save this as tests/high-rate-limit.json. The route under test uses the fixedWindow rule above (max: 50, window: "3s").

1
{
2
  "info": {
3
    "name": "high-rate-limit",
4
    "schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
5
  },
6
  "variable": [{ "key": "baseUrl", "value": "http://localhost:8080" }],
7
  "item": [
8
    {
9
      "name": "Test high rate limit",
10
      "request": {
11
        "method": "GET",
12
        "header": [{ "key": "Accept", "value": "application/json" }],
13
        "url": "{{baseUrl}}/api/high-rate-limit"
14
      },
15
      "event": [
16
        {
17
          "listen": "test",
18
          "script": {
19
            "type": "text/javascript",
20
            "exec": [
21
              "pm.test('status matches iteration', () => pm.response.to.have.status(pm.info.iteration < 50 ? 200 : 429))"
22
            ]
23
          }
24
        }
25
      ]
26
    }
27
  ]
28
}

Start the app, then:

1
npx newman run tests/high-rate-limit.json -n 51

Expected output (timings vary):

1
newman
2

3
high-rate-limit
4

5
→ Test high rate limit
6
  ✓ status matches iteration
7
  … (50 passing 200s)
8

9
→ Test high rate limit
10
  ✓ status matches iteration   # 51st request, status 429
11

12
┌─────────────────────────┬──────────┬──────────┐
13
│                         │ executed │   failed │
14
├─────────────────────────┼──────────┼──────────┤
15
│              iterations │       51 │        0 │
16
│                requests │       51 │        0 │
17
│            test-scripts │       51 │        0 │
18
│      prerequest-scripts │        0 │        0 │
19
│              assertions │       51 │        0 │
20
├─────────────────────────┼──────────┼──────────┤
21
│              total run duration: 1.4s         │
22
└─────────────────────────┴──────────┴──────────┘

failed must be 0. If request 51 is still 200, the rule is DRY_RUN, the window reset between calls, or you are hitting a different instance than the one that holds the counter. If every request is 429, you inherited a hot key from a previous run; wait for the 3-second window or change the test characteristic.

The same collection runs from Node's test runner:

1
import assert from "node:assert/strict";
2
import { describe, it } from "node:test";
3
import { fileURLToPath } from "node:url";
4
import { promisify } from "node:util";
5
import { run } from "newman";
6

7
const newmanRun = promisify(run);
8

9
describe("rate limit", () => {
10
  it("denies the 51st request", async () => {
11
    const summary = await newmanRun({
12
      collection: fileURLToPath(
13
        new URL("./high-rate-limit.json", import.meta.url),
14
      ),
15
      iterationCount: 51,
16
    });
17
    assert.equal(summary.run.failures.length, 0);
18
  });
19
});

1
node --test

How do I run the same test with k6?

1
import http from "k6/http";
2
import { check } from "k6";
3

4
export const options = {
5
  scenarios: {
6
    functional: {
7
      executor: "shared-iterations",
8
      vus: 1,
9
      iterations: 51,
10
    },
11
  },
12
  thresholds: {
13
    checks: ["rate==1"],
14
  },
15
};
16

17
export default function () {
18
  const res = http.get("http://localhost:8080/api/high-rate-limit", {
19
    headers: { Accept: "application/json" },
20
  });
21
  const expected = __ITER < 50 ? 200 : 429;
22
  check(res, { "status matches iteration": (r) => r.status === expected });
23
}

1
k6 run tests/high-rate-limit.k6.js

Expected summary:

1
     ✓ status matches iteration
2

3
     checks.........................: 100.00% ✓ 51      ✗ 0
4
     http_reqs......................: 51      35/s
5
     iterations.....................: 51      35/s

checks at 100% and ✗ 0 is the pass. A bot rule is a single iteration with headers: { "User-Agent": "curl" } and expected = 403.

Keep functional security tests in CI on every change to rules or to the handler that calls protect(). A refactor that moves protect() below a database write will still pass a mocked unit test and fail this suite, which is the point.

Wire the suite so the server is up before Newman or k6 start. A node --test file can spawn node --import tsx src/server.ts, wait on /health, then call newmanRun. GitHub Actions should use a single job: install, start the app in the background, wait, run npx newman run … -n 51 and k6 run if you keep both. Do not hit a shared staging site from every PR. You will collide on the same rate-limit key and flake.

Key the test route on a characteristic you control (x-test-run-id mapped to characteristics: ["header:x-test-run-id"], or a dedicated ARCJET_KEY site) so parallel CI jobs do not share a window. The 3-second window in the example is for the suite, not for production. Production login limits stay at minutes; the test route can be a /api/__test/rate-limit handler that you disable when NODE_ENV === "production".