Skip to content
Inbox Theory

AI & Crawlers

Robots.txt AI Policy

Checks robots.txt against a curated list of known AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, and more) and reports what each is and isn't allowed to access.

Checks robots.txt against a curated list of known AI crawlers — necessarily a snapshot, since new crawlers appear and existing ones get renamed regularly. A crawler can always choose to ignore robots.txt; this only reports what a well-behaved one would see.