TheyfetchUnderstand your data
Know who is fetching
A live answer, a search index, and model training are different reasons to visit your site. Theyfetch keeps those purposes separate.
Different bots, different jobs
- Live answers
- An assistant fetches a page while responding to a person’s question.
- Search indexing
- A crawler builds an index the assistant may search later.
- Training
- A crawler collects content for model development, separate from a live answer.
The catalog
27 of 27 bots
| Bot | Provider | Purpose | Verification method |
|---|---|---|---|
| claude-code | Anthropic | Coding agents | Unverifiable |
| Claude-SearchBot | Anthropic | Search index | IP or reverse DNS |
| Claude-User | Anthropic | Live answers | IP or reverse DNS |
| ClaudeBot | Anthropic | Training | IP or reverse DNS |
| ChatGPT-User | OpenAI | Live answers | IP or reverse DNS |
| OAI-SearchBot | OpenAI | Search index | IP or reverse DNS |
| GPTBot | OpenAI | Training | IP or reverse DNS |
| Perplexity-User | Perplexity | Live answers | IP or reverse DNS |
| PerplexityBot | Perplexity | Search index | IP or reverse DNS |
| Google-GeminiNotebook | Live answers | IP or reverse DNS | |
| Google-Agent | Live answers | IP or reverse DNS | |
| Googlebot | Search index | IP or reverse DNS | |
| bingbot | Microsoft | Search index | IP or reverse DNS |
| meta-externalfetcher | Meta | Live answers | Unverified |
| meta-webindexer | Meta | Search index | Unverified |
| meta-externalagent | Meta | Training | Unverified |
| MistralAI-User | Mistral | Live answers | IP or reverse DNS |
| MistralAI-Index | Mistral | Search index | IP or reverse DNS |
| MistralAI-Training | Mistral | Training | Unverified |
| Amzn-User | Amazon | Live answers | IP or reverse DNS |
| Amzn-SearchBot | Amazon | Search index | IP or reverse DNS |
| Amazonbot | Amazon | Training | IP or reverse DNS |
| DuckAssistBot | DuckDuckGo | Live answers | IP or reverse DNS |
| Applebot | Apple | Search index | IP or reverse DNS |
| CCBot | Common Crawl | Training | IP or reverse DNS |
| Bytespider | ByteDance | Training | Unverified |
| opencode | OpenCode | Coding agents | Unverifiable |
How verification works
We match the user agent against the catalog, then check the request IP against the operator’s published ranges or reverse-DNS rules when available. The IP and full user-agent string are discarded before analytics storage.
Matching a name alone does not prove who sent a request. Coding-agent fetches often come from a user’s own address, so they remain unverifiable.