How to Block Claude-User

Complete guide to blocking Claude-User (Anthropic) from crawling your website using robots.txt, server configuration, and Switch workflows.

Operated by AnthropicAI Assistants

Should You Block Claude-User?

Blocking Claude-User prevents your content from appearing in Anthropic's AI-generated answers. Each visit from this agent represents a real user asking about your content.

Consider allowing Claude-User for visibility, or use Switch to serve agent-optimized markdown content instead of blocking entirely.

Blocking Methods

1robots.txt

High for cooperative crawlers

Add a Disallow rule for Claude-User's user-agent string in your robots.txt file. This is the standard, cooperative method that well-behaved crawlers respect.

2Server-side UA filtering

High

Configure your web server (nginx, Apache, Cloudflare) to reject requests matching Claude-User's user-agent patterns. This blocks at the network level before your application processes the request.

3Switch Journey Workflows

Highest — granular, real-time control

Create a custom journey in Switch that detects Claude-User and routes it to a block action, challenge, redirect, or modified content — without touching your server configuration.

robots.txt — Block Claude-User

Add the following to your robots.txt file (at the root of your domain) to block Claude-User:

User-agent: Claude-User
Disallow: /

robots.txt — Allow with Restrictions

Alternatively, allow Claude-User on most pages while blocking specific directories:

User-agent: Claude-User
Disallow: /private/
Allow: /

Claude-User User-Agent Strings

Use these patterns to identify Claude-User in your server logs or firewall rules:

Claude-User

Frequently Asked Questions

Does blocking Claude-User affect my Google search rankings?

No. Blocking Claude-User does not affect your Google search rankings. Only blocking Googlebot impacts Google Search visibility.

Does Claude-User respect robots.txt?

Yes, Claude-User respects robots.txt directives. Adding a Disallow rule for its user-agent will prevent it from crawling blocked paths.

Can I allow Claude-User on some pages but not others?

Yes. Use robots.txt to disallow specific directories, or use Switch journey workflows for granular page-level control with conditional logic.

Go beyond robots.txt

Switch detects Claude-User in real-time and lets you build custom journey workflows — block, challenge, redirect, or serve modified content. No server changes required.

Get Started Free
Claude-User Profile Agents Directory