Crawlability

Crawlability shows which AI bots are allowed or blocked from accessing a website based on its robots.txt file. Simply select a tracked domain to see the results instantly, with no additional setup or account connection required. Peec AI checks the domain’s robots.txt against 40+ AI bots from 20+ vendors and shows the crawlability status for each one. Bot categories are based on publicly available information about their intended purpose. As more information about how these bots operate becomes available, Peec may update their classifications to keep them accurate.

​

Crawlability table

The table breaks down the status for each bot:

• Bot: The user-agent identifier (e.g., GPTBot, ClaudeBot).
• Platform: The AI vendor behind the bot (e.g., OpenAI, Anthropic, Google).
• Bot type: Training, Search, User Query, and Other.
• Status: Allowed, Partial, or Blocked.
• Reason: How the status was determined: explicit rules for this bot, or inherited from the global wildcard (*) rules.

​

URL tester

Enter any URL from your domain to check which AI bots are allowed or blocked by its robots.txt rules. Select a URL from your domain to see which AI bots are allowed or blocked from crawling it. Use these insights to identify any bots that may be unintentionally blocked and update your robots.txt rules where needed.

​

Interpreting crawlability

If an AI bot is blocked, it can’t crawl or access the content on that page, which may prevent your site from being used as a source in its responses. Use Crawlability to:

• Identify accidental blocks that could limit your visibility in AI responses.
• See which AI bots can access your content and which are restricted.
• Confirm robots.txt changes to make sure your updated rules are working as intended.

​

Bots

Bots show which AI bots are visiting your pages, which vendors they belong to, and the type of each bot.