Skip to main content
Crawlability shows you which AI bots are allowed or blocked by a site’s robots.txt file. Select a tracked domain and get results instantly with no setup or account connection required. Peec checks the domain’s robots.txt against 40+ AI bots from 20+ vendors, then shows you the status for each. We categorize bots based on publicly available data and their stated purpose. As real-world behavior becomes clearer, categories may be refined to ensure accuracy.

​
Crawlability table

The table breaks down the status for each individual bot:
  • Bot: the user-agent identifier (e.g., GPTBot, ClaudeBot).
  • Platform: the AI vendor behind the bot (e.g., OpenAI, Anthropic, Google)
  • Bot type: Training, Search, User Query, and Other
  • Status: Allowed, Partial, or Blocked
  • Reason: How the status was determined: explicit rules for this bot, or inherited from the global wildcard (*) rules
Use the in-table search bar to find a specific bot, or filter by platform, bot type, or status using the filters at the top.

​
URL Tester

Crawlability URL Tester
Here you can enter any URL to see which AI bots are allowed or blocked by your domain’s robots.txt rules. Simply choose a URL from your domain to analyze and see which bots are allowed or blocked from crawling it. You can then use this insight to troubleshoot if bots are unintentionally blocked or to enable crawlability for it.

​
Interpreting Crawlability

If a bot is blocked, it can’t access your content. This means it can’t use your site as a source in its responses. Use Crawlability to:
  • Catch accidental blocking before it affects your AI visibility
  • Understand which AI ecosystems can and can’t access your content
  • Verify that changes to your robots.txt are working as expected

​
Bots

The Bots show you which specific bots from which vendors are accessing and visiting your pages, and the type of each.