Back to the laboratory

On a fixed panel of 40 established sites1, access broadly held for the two crawler identities we tested after the September 15 policy transition. Among the 19 sites that showed a Cloudflare header in the period before, allowed requests rose from 49.6% to 56.2% for the ClaudeBot string and from 42.3% to 45.7% for GPTBot. Three sites moved by at least 30 percentage points overall, two toward allowing and one toward blocking.

The wider gap is between written rules and actual responses. Among requests our robots.txt evaluator classed as allowed, 31.8% before and 33.1% after were challenged or blocked for the ClaudeBot string. For GPTBot the shares were 30.8% and 32.1%. A browser string sent from the same address was gated on 20.8% and 21.7%, so part of that gap comes from the address or the client and not from the crawler’s name.

The period before ran August 15 to September 14, and the period after September 16 to October 15. The series holds 466 completed runs from one US vantage through November 13, including all 20 scheduled runs from September 13 to 17. The requests carried the published identity strings2 and came from our own address.

HasData measured 10,894 domains in two waves, including live enforcement on a subset. Our panel adds repeated observations of the same paths around the date. HasData’s dataset

Technical readout · Data card

Footnotes

  1. The panel covers established sites and the paths sampled. ↩

  2. They are not traffic from the named companies. ↩