Search EnginesActive

Claude-SearchBot

Anthropic's search indexing crawler for Claude's search features.

Operated by AnthropicOfficial docs

What is Claude-SearchBot?

Claude-SearchBot is Anthropic's dedicated search indexing crawler, separate from ClaudeBot which collects training data. Claude-SearchBot indexes content specifically to power Claude's search capabilities, enabling the AI assistant to find and cite relevant web pages when users ask questions.

This is a relatively new crawler that represents the emerging category of AI-native search engines. Unlike traditional search crawlers that feed into a search results page, Claude-SearchBot indexes content for retrieval-augmented generation (RAG), meaning your content gets cited directly in Claude's responses.

As Anthropic expands Claude's web search features, Claude-SearchBot's crawl volume is expected to increase. Being indexed by Claude-SearchBot means your content can appear as a cited source when millions of Claude users ask relevant questions.

User-Agent Strings

These are the known user-agent patterns used by Claude-SearchBot. Use them to identify this crawler in your server logs or configure robots.txt rules.

Claude-SearchBot

robots.txt example:

User-agent: Claude-SearchBot
Disallow: /private/
Allow: /

How to Manage Claude-SearchBot

1

Allow Claude-SearchBot if you want your content cited in Claude AI responses.

2

This is separate from ClaudeBot — you can allow search indexing while blocking training data collection.

3

Use Switch markdown journeys to serve well-structured content for better citation.

4

Monitor for increasing crawl volumes as Claude search features expand.

How to block Claude-SearchBot

Start managing Claude-SearchBot today

Switch detects, tracks, and lets you build custom journeys for Claude-SearchBot and 35+ other AI agents and crawlers. Set up in five minutes.

Get Started Free

Related Agents

Claude Computer Use

Browser Agents

Anthropic

Anthropic's Claude agent that controls a real browser to complete tasks.

Claude-User

AI Assistants

Anthropic

Anthropic's real-time web agent for when Claude users browse live content.

ClaudeBot

Commercial Crawlers

Anthropic

Anthropic's web crawler collecting training data for Claude models.

Applebot

Search Engines

Apple

Apple's search crawler powering Siri, Spotlight, and Safari Suggestions.

Baiduspider

Search Engines

Baidu

Baidu's web crawler serving the dominant search engine in China.

Bingbot

Search Engines

Microsoft

Microsoft Bing's search crawler, also powering Copilot AI answers.

Back to Agents Directory