> ## Documentation Index
> Fetch the complete documentation index at: https://docs.peec.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# AI bot types — training, indexing, and retrieval bots do different jobs

> Peec's server-log guide (Jun 2026) — GPTBot trains models, OAI-SearchBot indexes for AI search, ChatGPT-User fetches pages live per user request; blocking the wrong one solves the wrong problem.

AI bots are three different species, and each maps to a different stage of AI visibility. Peec's server-log guide ([full guide](https://peec.ai/blog/why-server-logs-are-crucial-for-ai-search-strategy), Jun 2026) distinguishes:

* **Training bots (e.g. GPTBot, Common Crawl)** crawl to collect model training data — they shape what the model knows by default. Common Crawl alone holds 300+ billion pages since 2008, and over 80% of GPT-3's training data came from a filtered version of it; it tags pages with PageRank-style importance scores. Blocked training bots mean absence from the model's baseline knowledge, and gaps cannot be fixed retroactively — training data only refreshes at retraining.
* **Search/indexing bots (e.g. OAI-SearchBot)** index content so it can be retrieved and cited in AI search. Blocked indexing bots mean your content never enters the retrieval pool, regardless of quality.
* **User-query/retrieval bots (e.g. ChatGPT-User)** fetch a specific page in real time because a user asked — request-driven, the highest-intent signal of the three.

Unblocking one does not unblock the others: `OAI-SearchBot` (search) and `GPTBot` (training) are separate user-agents with separate robots.txt rules — Deep Research obeys OAI-SearchBot specifically (see [How Deep Research reads your site](/research/how-deep-research-reads-your-site)). The healthiest log pattern is all three bot types present. How to act on the log data: [Server logs for AI search](/research/server-logs-for-ai-search).

**Source:** [Why server logs are crucial for AI search strategy](https://peec.ai/blog/why-server-logs-are-crucial-for-ai-search-strategy), Peec (Tomek Rudzki), Jun 2026.