Back to the laboratory

This is a breadth panel of 40 established sites, with 86.3% observed coverage on the dominant vantage on a US cloud host, one US egress address, and four declared User-Agent strings from an unverified sender. It cannot measure verified crawler access, identify which sites changed their settings, or test the new Search/Agent split directly. There is no pre-change sample for those identities. Each recorded outcome is for one public path, not a domain-wide policy.

What changed

On the sites with a Cloudflare header observed in the before period, the two named crawler strings were more often allowed in the after period: ClaudeBot rose from 49.6% to 56.2% and GPTBot from 42.3% to 45.7%. There is no broad decline in this panel to explain. Site outcomes diverged. Cloudflare’s final transition guidance described migration of existing preferences and distinct presets for newly onboarded domains. The established-site panel is not representative of new ad-supported domains.

Declared identity Cloudflare signal before Cloudflare signal after Others before Others after
Browser 1,505 / 2,204 (68.3%) 1,311 / 1,786 (73.4%) 1,855 / 2,436 (76.1%) 1,421 / 1,974 (72.0%)
ClaudeBot 1,093 / 2,204 (49.6%) 1,004 / 1,786 (56.2%) 1,329 / 2,436 (54.6%) 1,169 / 1,974 (59.2%)
GPTBot 933 / 2,204 (42.3%) 816 / 1,786 (45.7%) 1,532 / 2,436 (62.9%) 1,263 / 1,974 (64.0%)
python-httpx 1,276 / 2,204 (57.9%) 1,034 / 1,786 (57.9%) 1,739 / 2,436 (71.4%) 1,324 / 1,974 (67.1%)

Fixed groups: 19 sites showing a Cloudflare header on any before-period probe, 21 others. The numerator is allowed; the denominator includes every observed outcome including transport errors and auth redirects. The before window is August 15 to September 14 inclusive (116 runs), the after window September 16 to October 15 inclusive (94 runs). September 15 is excluded from both windows.

Robots policy and observed edge behavior

Among probes classified allow by the robots evaluator, about one third of probes with the ClaudeBot and GPTBot strings were challenged or hard blocked. A browser string on the same address also saw gating on about one fifth of such requests. The result does not apportion causes between the address, HTTP client, identity, infrastructure and site policy.

Declared identity Before: challenged or hard blocked / robots allow After: challenged or hard blocked / robots allow
Browser 780 / 3,746 (20.8%) 651 / 2,999 (21.7%)
ClaudeBot 969 / 3,050 (31.8%) 870 / 2,626 (33.1%)
GPTBot 1,084 / 3,514 (30.8%) 964 / 3,002 (32.1%)
python-httpx 1,092 / 3,746 (29.2%) 1,009 / 2,999 (33.6%)

no_rule, no_robots and disallow are excluded from this denominator. allow includes the collector’s default allow in a matching readable group. Outcome and robots policy are separate measurements.

Site changes of at least 30 points

Pooling all four identities within each site, three sites crossed the threshold: Amazon 265/464 to 376/376 (up 42.9 points), GitLab 242/464 to 318/376 (up 32.4 points), and NYTimes 232/464 to 19/376 (down 44.9 points). At the individual identity level, 12 site/identity pairs across eight sites crossed the threshold. Each shift is from our address, for the paths sampled. These are observed shifts; they do not identify the policy that changed.

Coverage

UTC month Complete runs at dominant vantage Gaps over 7.5 hours between finishes, assigned to the later month
July 1 to 31 121 1
August 122 1
September 103 11
October 96 21
November 1 to 13 24 11

466 runs used the IAD vantage tag (US). Two other complete runs from the same address carried an EWR tag and are excluded from comparisons. From first to last dominant-vantage start there are 540 nominal six-hour slots, for 86.3% observed coverage. The 45 finish gaps include 76 estimated missed slots using nearest six-hour start intervals; two additional starts do not fill the missing slots. The longest interval was 54.0 hours (November 8 to 10). All 20 nominal runs from September 13 to 17 were present. Collection was interrupted by a storage issue on November 14.

Reproduction and context

The queries are available on request. The November summary ends before November 14 UTC and keeps the four strings used in the September comparison.

HasData’s two-wave dataset is larger and includes live enforcement both before and after. Cloudflare Radar reports inside-network aggregates. This panel adds repeated observations on the same small panel around the date and a per-request pairing of robots outcome and edge behavior.

Lab note · Data card