A scraping gate must cover the data routes

A protected page can fetch its data from an unprotected endpoint. Checking the HTML navigation alone leaves that endpoint outside the gate. Heretic's reference Cloudflare Worker instead checks every routed request unless its path appears in an explicit public set.

We tested that boundary in workerd. The experiment records both the response status and whether the gate contacted a synthetic origin. A redirect is useful evidence only if the protected request did not already reach the application.

Ten requests, with origin calls counted

The fixture uses Miniflare 5.20260811.1-alpha and Node 22.22.2. It loads the actual reference gate and clearance verifier, generates an ephemeral Ed25519 signer and supplies a synthetic public-key response. Every outbound fetch is intercepted. Nothing is sent to a production origin or Heretic's API.

The cases ran sequentially in one Worker instance on 17 October 2026. Counts below are per request, not cumulative. The address used for comparison is injected by the test.

The image header and filename suffix never admit a request. The admitted API request is the control: the same data route can reach the origin when the gate's conditions hold. /robots.txt is a second control because the reference configuration deliberately makes it public.

Put the decision before the origin fetch

The relevant ordering starts with a small, explicit set:

const PUBLIC_PATHS = new Set([
  '/robots.txt',
  '/favicon.ico',
  '/.well-known/heretic-site-verification',
]);

For a path outside that set, the cookie branch verifies the clearance and then evaluates the customer's admission rule:

const checked = await verifyClearance(cookie, options);
if (checked.valid) {
  return allowed(checked.claims, request) ? fetch(request) : refuse();
}
if (checked.reason === 'jwks') return unavailable();

Those two conditions establish different facts. Verification authenticates the signed report under the configured contract. The reference allowed function compares the selected device address with the request address, then checks the verdict or ceremony outcome. A customer can replace that policy. Our denied-verdict case uses a correctly signed report and still receives 403.

When no usable clearance cookie is present, ordinary GET and HEAD requests enter the redirect flow. Other ordinary methods return 403. A verified cookie that fails the admission policy returns 403 directly, including for GET. Callback routes have their own validation branches. The tested POST body is never forwarded or replayed through a redirect. An application using background verification must complete that flow before submitting the protected operation again.

The key-fetch failure is distinct. We first exercise a known signing key, then send a token naming an unknown key and make the synthetic key endpoint return 503. That forces a lookup instead of merely reusing the cached known key. The gate returns 503 with zero origin calls. This case says nothing about whether already cached keys remain usable during an outage.

Destination metadata does not grant clearance

The Fetch Metadata specification defines Sec-Fetch-Dest as the request's intended destination. In a browser, an image request can carry image. That describes how the response is to be used; it is not an assertion that a Heretic check passed.

The Sec- prefix prevents ordinary browser script from setting these headers. Our fixture submits them directly to workerd. It therefore tests the server's admission behavior, not a browser-script bypass. The gate requires clearance regardless of the supplied destination.

The address comparison also has a deployment premise. This Worker reads CF-Connecting-IP in Cloudflare's request context. Cloudflare documents different behavior for Worker subrequests. Copying the comparison into a server that accepts an arbitrary client-supplied header would change that premise. This local test supplies the address deliberately and does not validate the production header path.

Route coverage remains a separate requirement

The experiment dispatches every request through the gate. It cannot prove that a deployed route configuration covers every API or export URL, or that an origin cannot be reached directly. Both must be checked in the deployment that uses it.

The useful assertion is narrower and testable: for a protected request handled by this gate, the origin receives no call until the clearance and admission checks succeed. Counting origin calls makes that assertion visible. A page that looks protected is not enough evidence that its data requests are protected too.

Reproduce the request matrix

Download the standalone fixture and recorded results, extract it, then run:

cd scraping-gate
npm ci
node matrix.mjs

The lockfile pins the test dependencies. Installation may download packages; the test intercepts Worker outbound requests. The output records each status, origin-call count and key-fetch count. The ten rows repeated during separate acceptance on 18 October.