How to Block SemrushBot

Complete guide to blocking SemrushBot (Semrush) from crawling your website using robots.txt, server configuration, and Switch workflows.

Operated by SemrushSEO Scrapers

Should You Block SemrushBot?

SemrushBot builds Semrush's commercial SEO database. Blocking it has no impact on your search rankings but may affect your site's data in Semrush's tools.

Blocking is reasonable to reduce server load, especially if you don't use Semrush's SEO tools.

Blocking Methods

1robots.txt

High for cooperative crawlers

Add a Disallow rule for SemrushBot's user-agent string in your robots.txt file. This is the standard, cooperative method that well-behaved crawlers respect.

2Server-side UA filtering

High

Configure your web server (nginx, Apache, Cloudflare) to reject requests matching SemrushBot's user-agent patterns. This blocks at the network level before your application processes the request.

3Switch Journey Workflows

Highest — granular, real-time control

Create a custom journey in Switch that detects SemrushBot and routes it to a block action, challenge, redirect, or modified content — without touching your server configuration.

robots.txt — Block SemrushBot

Add the following to your robots.txt file (at the root of your domain) to block SemrushBot:

User-agent: SemrushBot
Disallow: /

User-agent: semrushbot
Disallow: /

robots.txt — Allow with Restrictions

Alternatively, allow SemrushBot on most pages while blocking specific directories:

User-agent: SemrushBot
Disallow: /private/
Allow: /

User-agent: semrushbot
Disallow: /private/
Allow: /

SemrushBot User-Agent Strings

Use these patterns to identify SemrushBot in your server logs or firewall rules:

SemrushBot
semrushbot

Frequently Asked Questions

Does blocking SemrushBot affect my Google search rankings?

No. Blocking SemrushBot does not affect your Google search rankings. Only blocking Googlebot impacts Google Search visibility.

Does SemrushBot respect robots.txt?

Yes, SemrushBot respects robots.txt directives. Adding a Disallow rule for its user-agent will prevent it from crawling blocked paths.

Can I allow SemrushBot on some pages but not others?

Yes. Use robots.txt to disallow specific directories, or use Switch journey workflows for granular page-level control with conditional logic.

Go beyond robots.txt

Switch detects SemrushBot in real-time and lets you build custom journey workflows — block, challenge, redirect, or serve modified content. No server changes required.

Get Started Free
SemrushBot Profile Agents Directory