> ## Documentation Index
> Fetch the complete documentation index at: https://docs.peec.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Server logs for AI search — diagnose crawled-but-not-cited pages

> Peec's guide (Jun 2026) — server logs show whether AI bots can reach you and which pages they request; pages crawled often but rarely cited point to technical, quality, or prompt-tracking gaps.

Server logs are the most direct way to see why AI search visibility underperforms. Peec's guide ([full guide](https://peec.ai/blog/why-server-logs-are-crucial-for-ai-search-strategy), Jun 2026) sets the check order:

1. **Crawlability first.** A bot blocked in robots.txt cannot help you, no matter how good the page is. Rule access out before tuning anything downstream.
2. **All three bot types present** — training, search, and user-query bots each map to a different visibility stage; a missing type names the gap. See [AI bot types](/research/ai-bot-types).
3. **Cross crawls against citations.** The real insight comes from combining how often bots request a page with whether it shows up as a source.

A page crawled often but rarely cited points to one of three different problems, each with a different fix (Jun 2026 guidance):

* **Technical:** bots reach the page but heavy JavaScript blocks clean parsing.
* **Content quality:** accessible, but LLMs choose not to include it.
* **Prompt-tracking gap:** the page performs well on prompts you are not tracking — broaden coverage using the topics the crawled page covers.

Also check where bots land, not just how often: bots hitting only the homepage and top-level pages signal a crawl-depth or internal-linking issue — a page that never appears in logs cannot be fixed by content tuning.

**Source:** [Why server logs are crucial for AI search strategy](https://peec.ai/blog/why-server-logs-are-crucial-for-ai-search-strategy), Peec (Tomek Rudzki), Jun 2026.