Install the crawler probe

Install the probe

One package, zero runtime dependencies. site is the only required option; a missing token throws at init because that is a wiring error. The token is a routing key, not a secret — it appears in your source. The probe cannot check Origin (server-side callers do not send one), so the real limit is the daily cap: 2,000 probe events and 500 distinct paths on the free plan, ×10 on paid. When a cap is hit the console now says so instead of silently reporting a smaller number. Forged reports only pollute your own dashboard, and rotating the token stops them. Get it from Search → Sessions → Traffic.

npm install @visdenda/agent-watch

The same package also ships @visdenda/agent-watch/markdown for content negotiation — see the Markdown guide.

Fetch API runtimes

Everything that hands you a Request and expects a Response uses the same wrapper. Response-aware: the real status is reported after your handler returns. When the runtime passes a context with waitUntil, the POST rides it so it never delays the response; without one (Bun, Deno) the probe is a plain fire-and-forget request.

import { wrapFetch } from "@visdenda/agent-watch/workers";

export default {
  fetch: wrapFetch({ token: "vdc_your_token" }, (request, env, ctx) =>
    env.ASSETS.fetch(request),
  ),
};

Hono runs on all of those runtimes but has its own middleware signature — use contextMiddleware, which reports after await next() with the final status and reuses executionCtx.waitUntil under Workers:

import { Hono } from "hono";
import { contextMiddleware } from "@visdenda/agent-watch/hono";

const app = new Hono();
app.use("*", contextMiddleware({ token: "vdc_your_token" }));

Node servers

Node hands you a request and a response object, so the probe reports on the response's finish event — a 500 is never recorded as a 200.

import express from "express";
import { nodeMiddleware } from "@visdenda/agent-watch/node";

const app = express();
app.use(nodeMiddleware({ token: "vdc_your_token" }));

Fastify has its own lifecycle, but req.raw and reply.raw are standard Node objects, so one hook is enough — no separate package:

import { nodeMiddleware } from "@visdenda/agent-watch/node";

const probe = nodeMiddleware({ token: "vdc_your_token" });

fastify.addHook("onRequest", (request, reply, done) => {
  probe(request.raw, reply.raw, done);
});

Koa works through koa-connect (or by passing ctx.req / ctx.res yourself), and NestJS on its default Express platform takes the same middleware via app.use(...). There is deliberately no per-framework package to keep in sync.

Framework hooks

Astro, SvelteKit and SolidStart do not hand you a bare fetch handler — they run your code inside their own hook. The recipe is the same either way, and it is shorter than a package would be. Whether you get a status depends on the signature, not on the brand.

Astro — next() resolves to a Response, so the real status is yours. context.clientAddress supplies the first-hop IP.

// src/middleware.ts
import { defineMiddleware } from "astro:middleware";
import { createAgentWatch } from "@visdenda/agent-watch";

const watch = createAgentWatch({ token: "vdc_your_token" });

export const onRequest = defineMiddleware(async (context, next) => {
  const response = await next(); // MiddlewareNext 返回 Response
  watch.probe({
    ua: context.request.headers.get("user-agent"),
    path: new URL(context.request.url).pathname,
    status: response.status,
    ip: context.clientAddress,
  });
  return response;
});

SvelteKit — same story with handle: resolve(event) returns the response, and event.getClientAddress() gives the IP.

// src/hooks.server.ts
import { createAgentWatch } from "@visdenda/agent-watch";

const watch = createAgentWatch({ token: "vdc_your_token" });

export const handle = async ({ event, resolve }) => {
  const response = await resolve(event);
  watch.probe({
    ua: event.request.headers.get("user-agent"),
    path: event.url.pathname,
    status: response.status,
    ip: event.getClientAddress(),
  });
  return response;
};

SolidStart — createMiddleware takes H3 middleware now, and await next() resolves to the response (check instanceof Response first — it can also be a non-response body).

// src/middleware/index.ts
import { createMiddleware } from "@solidjs/start/middleware";
import { createAgentWatch } from "@visdenda/agent-watch";

const watch = createAgentWatch({ token: "vdc_your_token" });

export default createMiddleware([
  async (event, next) => {
    const response = await next();
    watch.probe({
      ua: event.request.headers.get("user-agent"),
      path: new URL(event.request.url).pathname,
      status: response instanceof Response ? response.status : undefined,
      ip: event.clientAddress,
    });
    return response;
  },
]);

Nuxt (Nitro) is different: defineEventHandler in server/middleware/ sees the request but not the response, so report without status — the same class as Next middleware, and the console shows no-status-source. Getting the status there means Nitro's afterResponse hook (or the response object on a Node preset), which is more moving parts than the signal is worth.

Next.js middleware

Request-only: the response does not exist yet, so these hits are stored without a status and the console shows no-status-source. Middleware on Vercel, Netlify or a self-hosted Node server is still server-side, so this is a Tier A install. The SDK still sends the x-agent-watch version header, so you can tell which sites are on which client build.

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";
import { requestMiddleware } from "@visdenda/agent-watch/next";

const probe = requestMiddleware({ token: "vdc_your_token" });

export function middleware(request: NextRequest) {
  probe(request);
  return NextResponse.next();
}

Probe v2 options

The defaults work without configuration. These options change what the probe can see, or what it tells you when a report fails; two endpoints exist for operations.

• trustProxy: true — only when the site really sits behind a reverse proxy. By default the probe trusts cf-connecting-ip and ignores x-forwarded-for, because any client can send that header and poison the ASN rollup.
• batch: { maxEvents: 20, flushMs: 5000 } — off by default. A buffer is lost when the isolate is recycled, so batching is a cost optimisation, not a correctness feature. Workers cannot rely on a timer after the response; the SDK keeps the invocation alive through waitUntil when one is available. Values are clamped to the server contract (maxEvents ≤ 50, flushMs ≤ 10 s) and a flush is split into ≤ 32 KB requests, so one long referrer cannot turn the whole batch into a rejected request.
• country / colo — per-call overrides for runtimes without a platform header. Country is read from cf-ipcountry, x-vercel-ip-country or Netlify's x-nf-geo; colo comes from cf-ray and is the edge node, not the visitor's country.
• skipBrowserNavigations: false — report human navigations too. Document navigations (sec-fetch-mode + sec-fetch-dest + an HTML accept) are skipped on every engine — Firefox and Safari send no Chromium client hints — while clicks from AI answers are reported when the referrer is a known AI host.
• sendIp: false — stop reporting the client IP. It is on by default because ASN evidence needs it; the IP is used for the ASN lookup and is never stored.
• onError / debug — failures are silent by default. onError receives { path, status?, error?, events? } for non-2xx responses and network/timeout errors; debug: true warns once with a pointer to watch.check(). When both are set, onError wins.

trustProxy, batch and country/colo ship in @visdenda/agent-watch 0.4.0; onError and debug ship in 0.4.1. skipBrowserNavigations and sendIp predate v2; 0.4.0 refines the navigation skip (engine-neutral detection plus AI referrers). Older 0.3.x probes still report; they just cannot send the v2 fields.

Cap tiers are per site per day: 2,000 reported events and 500 distinct page paths on the free plan, 20,000 and 5,000 on paid. Hitting a cap no longer hides the loss — the dashboard shows which numbers are a lower bound and how many path hits went to the overflow bucket. Machine files (/robots.txt, /llms.txt, sitemaps, .md) are counted separately from page crawls.

curl -X POST https://visdenda.com/v1/probe/check \
  -H 'content-type: application/json' \
  -d '{"site":"vdc_your_token"}'

/v1/probe/batch accepts 1–50 events per request (≤ 32 KB); /v1/probe/check answers {"data":{"valid":true}} only when the token is active and the site is verified — both return 204 / 200 without leaking site details. Call watch.check() to run the same check from your own process (watch.verify() is the boolean form), because a typo in the token otherwise looks like "installed, no error, zero data".

If you also use Markdown negotiation, wrap the probe outside the Markdown middleware so it sees the final text/markdown response; that is what fills neg. The middleware supports an optional response cache (cache: { ttlMs: 300000 }) on runtimes with caches; it stores only 200 responses, skips any response the origin marked as personalized, and the stored copy keeps Vary: Accept with max-age, private so an upstream CDN never serves the Markdown variant to browsers.

Any other stack

The probe protocol is HTTP + JSON, so Go, Rust, PHP, Python, Ruby and Java need no SDK: one POST per request is enough. status and ip are optional, and only known crawler user agents are stored — everything else is dropped server-side.

curl -X POST https://visdenda.com/v1/probe \
  -H 'content-type: application/json' \
  -d '{"site":"vdc_your_token","ua":"GPTBot/1.2","path":"/pricing","status":200,"ref":"https://chatgpt.com/c/abc","country":"US"}'

PHP has no SDK — the endpoint is the integration. Run it after the response is finished so the visitor never waits; behind a CDN, pass the real client IP header instead of REMOTE_ADDR.

<?php
// PHP-FPM, Laravel, Symfony, WordPress: one fire-and-forget POST per request.
$watch = static function (?int $status): void {
    $payload = [
        'site' => 'vdc_your_token',
        'ua'   => $_SERVER['HTTP_USER_AGENT'] ?? '',
        'path' => parse_url($_SERVER['REQUEST_URI'] ?? '/', PHP_URL_PATH) ?: '/',
    ];
    if ($status !== null) $payload['status'] = $status;
    if (!empty($_SERVER['HTTP_REFERER'])) $payload['ref'] = $_SERVER['HTTP_REFERER'];
    if (!empty($_SERVER['REMOTE_ADDR'])) $payload['ip'] = $_SERVER['REMOTE_ADDR'];

    $context = stream_context_create(['http' => [
        'method'        => 'POST',
        'header'        => "content-type: application/json\r\n",
        'content'       => json_encode($payload),
        'timeout'       => 1,
        'ignore_errors' => true,
    ]]);
    @file_get_contents('https://visdenda.com/v1/probe', false, $context);
};

// Finish the response first so the visitor never waits for the report.
register_shutdown_function(static function () use ($watch): void {
    if (function_exists('fastcgi_finish_request')) fastcgi_finish_request();
    $watch(http_response_code() ?: 200);
});

The contract, so a Java filter, a Go middleware or an Axum layer can implement it without guessing — they all have to capture the status themselves, because the response object is the only place it exists:

• POST /v1/probe, JSON, body ≤ 4 KB. Fields: site and ua (required, UA ≤ 512 chars), path (≤ 512 chars; query and hash stripped), status (100–599), ip (first hop only), ref (≤ 1024), utm (≤ 64), neg (html / markdown), dur (ms, ≤ 600000), country and colo.
• site is your site token, not your domain. A domain would let anyone post hits for any site, with no way to revoke a sender; the token is write-only, shared by the beacon and the probe, and rotating it stops forged writes.
• The answer is 204 whether the hit was stored or dropped (unknown user agent, rotated token). Never branch on it — check the console.
• Batch up to 50 events in one POST /v1/probe/batch (≤ 32 KB), and use POST /v1/probe/check to verify a token. Both are exempt from the same-origin guard like the single endpoint.
• Per site per day: 2,000 probe events and 500 distinct page paths on free (×10 on paid). Over the path cap the hits still count in an overflow bucket, and the console states that the list is a lower bound — nothing is dropped silently anymore.

On a home-grown JS/TS HTTP layer, skip the wrappers and report a parsed request directly — this is the escape hatch behind every adapter above:

import { createAgentWatch } from "@visdenda/agent-watch";

const watch = createAgentWatch({ token: "vdc_your_token" });

watch.probe({
  ua: request.headers["user-agent"],
  path: request.url,
  status: 200,
  ip: request.headers["x-forwarded-for"],
});

Edge logs: Cloudflare Logpush

Middleware only sees requests that reach the origin. A cached page, a static asset or a request answered at the edge never arrives, so the crawl picture has a hole exactly where agents are most active. Edge logs close it. Point a Cloudflare Logpush job at the collector and the same server-side classification, identity verification and daily rollups apply — no new SDK, no second dashboard.

{
  "name": "visdenda-ai-crawlers",
  "dataset": "http_requests",
  "destination_conf": "https://visdenda.com/v1/probe/logs?site=vdc_your_token",
  "output_options": {
    "field_names": [
      "RayID",
      "EdgeStartTimestamp",
      "ClientIP",
      "ClientRequestHost",
      "ClientRequestMethod",
      "ClientRequestURI",
      "ClientRequestUserAgent",
      "ClientRequestReferer",
      "EdgeResponseStatus",
      "ClientCountry",
      "ColoCode"
    ],
    "max_upload_records": 500
  }
}

The token sits in the destination URL because Logpush cannot set custom headers. It is the same write-only site token as the probe: rotating it stops a leaked URL. Keep max_upload_records at 500 or lower — a bigger batch is rejected with 413, and Logpush retries a rejected batch.

Field names are the official http_requests ones; the collector maps them, so the job above is the whole integration:

• RayID → dedupe id. Re-delivery of the same batch is counted once; ids are kept for 7 days, the same window a backfill may cover.
• EdgeStartTimestamp → event time. Events are grouped by UTC day from the timestamp, not from arrival, so a backfill lands on the right day.
• ClientRequestHost + ClientRequestURI → path (query stripped), EdgeResponseStatus → ok / notfound / blocked / error.
• ClientRequestUserAgent is matched server-side; non-crawler rows are dropped. ClientIP is used for vendor IP-range and reverse-DNS verification and is never stored.

Another CDN or a log warehouse can post the canonical shape instead. One request carries up to 500 events, and every event needs an id (or a stable one is derived) so replays are safe:

curl -X POST "https://visdenda.com/v1/probe/logs?site=vdc_your_token" \
  -H 'content-type: application/json' \
  -d '{"events":[
    {"id":"req-1","ts":"2026-11-12T09:00:00Z","url":"https://your-site.example/pricing","status":200,"ua":"GPTBot/1.2","ip":"20.171.207.1"}
  ]}'

Log ingestion has its own daily budget: 10,000 events and 500 stored rows on the Free plan, 50,000 events and 3,000 rows on Pro. Rows, not raw logs, are what gets stored, so the budget is the cost ceiling; events beyond it fold into overflow rows — totals stay right while per-path detail degrades. A global daily ceiling protects the platform regardless of how many sites connect. The free probe caps (2,000 events / 500 paths) still apply to per-request reports.

What each setup can see

Do not describe a script-only install as "you can see AI crawlers". The crawler block answers na with a pointer here until a server probe is installed.

Want to know what a crawler is before you allow it? Read the crawler directory — robots.txt token, documented user agent and vendor source for all 63 entries.

Check it worked

Curl your own site with a crawler user agent. Within a minute the Crawlers block names the bot and the path you hit; statuses feed the invalid-crawl classification. A wrong site token or an unknown user agent is dropped — the endpoint still answers 204, so check the console, not the HTTP response.

curl -A "GPTBot/1.2 (+https://openai.com/gptbot)" \
  https://your-site.example/pricing

Sending a real bot string? The GPTBot entry has the documented user agent and reverse-DNS suffix, so you can tell a live probe from a spoofed header.

Prefer the raw script-tag path? The first-party telemetry guide covers page views and vitals; this page covers the crawler sensor.