Connecting your data
To get started, go to Crawl Insights and connect a data source. We currently support two options:- Cloudflare Workers: for continuous, real-time log ingestion if your site runs on Cloudflare.
- CSV/CLF File upload: upload a server log file directly to analyze a specific time period.
CSV/CLF files must have a .csv, .clf, or .txt extension.
Dashboard Overview
Once the data is loaded, your dashboard will appear. Here, you can access key information and use different tools to analyze your data.
- Filters: Date, Platform (e.g., OpenAI, Google), Bot type (Training, Search, User Query, Other), and Bot (e.g., GPTBot, ClaudeBot)
- KPI Summary: A row of key metrics at the top gives you a snapshot of your filtered data based on total bot visits, active bots, and failure rate (e.g., proportion of requests that returned an error (4xx or 5xx))
- Crawl activity over time: The line chart shows AI bot visits, with a separate line for each AI model. Switch between hourly (only visible in a 3-day view), daily, weekly, and monthly views - daily is the default. Use this to spot trends, sudden spikes, or unexplained drops in bot activity.
- Content section: Further down, you’ll see AI bot traffic broken down by folder and URL, showing which parts of your site AI bots visit most.
Crawl breakdown
The crawl breakdown will show you a set of bar charts that break down bot visits by different dimensions.
- Platform: the AI vendor (e.g., OpenAI, Anthropic, Google)
- Bot: the specific user agent (e.g., GPTBot, ClaudeBot, PerplexityBot)
- Bot type: Training / Search / User query / Other
The Bot type indicates the purpose of the crawl:
- Training bots collect data to build or refine AI models.
- Search bots browse the web to find up-to-date information when needed.
- User Query bots access your website to fetch content on behalf of a user.
Insight Details
Insight Details provides a URL-level breakdown of AI bot activity and source performance. You can access it from the in-page navigation bar or by clicking the See Details button above the By URL chart. In here you will see:- Filters: The same global filters apply (date range, platform, bot type, bot). Within the table, you can also filter by folder (drill into a specific section of your URL structure) and status code (show only URLs that returned a specific HTTP response (e.g., 200, 404, 500)). You can also use the table’s search bar to quickly find a specific URL.
- KPI summary:
- Total bot visits: Total requests across all URLs in the selected filters
- Active bots: Number of distinct bots active in this view
- Failure rate: Proportion of requests that returned an error (4xx or 5xx status code)
- Top-visited folder: The top-level section of your site (e.g., /blog/, /products/) receiving the most bot visits
- Top-visited URL: The single URL receiving the most bot visits
Visited URLs table
This table shows the pages on your site that can be crawled and actively indexed, with detailed information for each. This will give you a quick overview of which pages are receiving bot visits, how often they occur, and how this differs across all bots currently visiting that page. You will see:- URL: The requested page path from your domain
- Folder: The primary section or directory the URL belongs to
- Bot visits: Total AI bot visits to this URL in the selected period
- Platforms: Bot visits by the vendor behind the bots requesting the URL
- Status codes: The HTTP response the server returned to bots
- Retrievals: The sum total of how many times this URL appeared as a source in AI chat responses
- Citation rate: Average number of inline citations per chat when this URL is retrieved as a source
- Topics: How many of your tracked topics had prompts where this page appeared as a source
Prompt data comes from the prompts you are tracking. It is not influenced by AI bot activity.
Status Codes
Status codes show how your server responded to AI bot visits, based on your connected log data. The overview table always shows a baseline set of status codes, even if they have zero visits in your log data. Additional codes appear only when present in your logs.| Status code | Description |
|---|---|
| 200 | OK |
| 301 | Moved Permanently |
| 302 | Found (Temporary Redirect) |
| 304 | Not Modified |
| 307 | Temporary Redirect |
| 308 | Permanent Redirect |
| 401 | Unauthorized |
| 403 | Forbidden |
| 404 | Not Found |
| 410 | Gone |
| 429 | Too Many Requests |
| 500 | Internal Server Error |
| 502 | Bad Gateway |
| 503 | Service unavailable |
| 504 | Gateway Timeout |
FAQs
How do I connect?
Go to Crawl Insights and follow the setup prompt. The method depends on your hosting setup:- If you use Cloudflare, you’ll connect via a Cloudflare Workers, which streams your logs to Peec in real time.
- If you use another provider or prefer to upload a specific dataset, use the CSV or CLF file upload option — just drag and drop your log file or select it from your file picker.