How to Block ClaudeBot

Complete guide to blocking ClaudeBot (Anthropic) from crawling your website using robots.txt, server configuration, and Switch workflows.

Operated by AnthropicCommercial Crawlers

Should You Block ClaudeBot?

ClaudeBot collects data for AI model training. Blocking it prevents your content from being used in Anthropic's AI products without affecting your search visibility.

This is a common and recommended action for sites that want to control how their content is used in AI training.

Blocking Methods

1robots.txt

High for cooperative crawlers

Add a Disallow rule for ClaudeBot's user-agent string in your robots.txt file. This is the standard, cooperative method that well-behaved crawlers respect.

2Server-side UA filtering

High

Configure your web server (nginx, Apache, Cloudflare) to reject requests matching ClaudeBot's user-agent patterns. This blocks at the network level before your application processes the request.

3Switch Journey Workflows

Highest — granular, real-time control

Create a custom journey in Switch that detects ClaudeBot and routes it to a block action, challenge, redirect, or modified content — without touching your server configuration.

robots.txt — Block ClaudeBot

Add the following to your robots.txt file (at the root of your domain) to block ClaudeBot:

User-agent: ClaudeBot
Disallow: /

User-agent: claudebot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Anthropic
Disallow: /

robots.txt — Allow with Restrictions

Alternatively, allow ClaudeBot on most pages while blocking specific directories:

User-agent: ClaudeBot
Disallow: /private/
Allow: /

User-agent: claudebot
Disallow: /private/
Allow: /

User-agent: Claude-Web
Disallow: /private/
Allow: /

User-agent: anthropic-ai
Disallow: /private/
Allow: /

User-agent: Anthropic
Disallow: /private/
Allow: /

ClaudeBot User-Agent Strings

Use these patterns to identify ClaudeBot in your server logs or firewall rules:

ClaudeBot
claudebot
Claude-Web
anthropic-ai
Anthropic

Frequently Asked Questions

Does blocking ClaudeBot affect my Google search rankings?

No. Blocking ClaudeBot does not affect your Google search rankings. Only blocking Googlebot impacts Google Search visibility.

Does ClaudeBot respect robots.txt?

Yes, ClaudeBot respects robots.txt directives. Adding a Disallow rule for its user-agent will prevent it from crawling blocked paths.

Can I allow ClaudeBot on some pages but not others?

Yes. Use robots.txt to disallow specific directories, or use Switch journey workflows for granular page-level control with conditional logic.

Go beyond robots.txt

Switch detects ClaudeBot in real-time and lets you build custom journey workflows — block, challenge, redirect, or serve modified content. No server changes required.

Get Started Free
ClaudeBot Profile Agents Directory