How to Block Bytespider
Complete guide to blocking Bytespider (ByteDance) from crawling your website using robots.txt, server configuration, and Switch workflows.
Should You Block Bytespider?
Bytespider collects data for AI model training. Blocking it prevents your content from being used in ByteDance's AI products without affecting your search visibility.
This is a common and recommended action for sites that want to control how their content is used in AI training.
Blocking Methods
1robots.txt
High for cooperative crawlersAdd a Disallow rule for Bytespider's user-agent string in your robots.txt file. This is the standard, cooperative method that well-behaved crawlers respect.
2Server-side UA filtering
HighConfigure your web server (nginx, Apache, Cloudflare) to reject requests matching Bytespider's user-agent patterns. This blocks at the network level before your application processes the request.
3Switch Journey Workflows
Highest — granular, real-time controlCreate a custom journey in Switch that detects Bytespider and routes it to a block action, challenge, redirect, or modified content — without touching your server configuration.
robots.txt — Block Bytespider
Add the following to your robots.txt file (at the root of your domain) to block Bytespider:
User-agent: Bytespider Disallow: / User-agent: bytespider Disallow: / User-agent: Bytedance Disallow: /
robots.txt — Allow with Restrictions
Alternatively, allow Bytespider on most pages while blocking specific directories:
User-agent: Bytespider Disallow: /private/ Allow: / User-agent: bytespider Disallow: /private/ Allow: / User-agent: Bytedance Disallow: /private/ Allow: /
Bytespider User-Agent Strings
Use these patterns to identify Bytespider in your server logs or firewall rules:
Frequently Asked Questions
Does blocking Bytespider affect my Google search rankings?
No. Blocking Bytespider does not affect your Google search rankings. Only blocking Googlebot impacts Google Search visibility.
Does Bytespider respect robots.txt?
Yes, Bytespider respects robots.txt directives. Adding a Disallow rule for its user-agent will prevent it from crawling blocked paths.
Can I allow Bytespider on some pages but not others?
Yes. Use robots.txt to disallow specific directories, or use Switch journey workflows for granular page-level control with conditional logic.
Go beyond robots.txt
Switch detects Bytespider in real-time and lets you build custom journey workflows — block, challenge, redirect, or serve modified content. No server changes required.
Get Started Free