How to Block OAI-SearchBot

Complete guide to blocking OAI-SearchBot (OpenAI) from crawling your website using robots.txt, server configuration, and Switch workflows.

Operated by OpenAISearch Engines

Should You Block OAI-SearchBot?

Caution: OAI-SearchBot is a search engine crawler. Blocking it will remove your pages from OpenAI's search index, which directly impacts your organic traffic and visibility.

Only block OAI-SearchBot if you intentionally want to de-index your site from OpenAI. Instead, consider using Switch to serve optimized content or manage specific page access.

Blocking Methods

1robots.txt

High for cooperative crawlers

Add a Disallow rule for OAI-SearchBot's user-agent string in your robots.txt file. This is the standard, cooperative method that well-behaved crawlers respect.

2Server-side UA filtering

High

Configure your web server (nginx, Apache, Cloudflare) to reject requests matching OAI-SearchBot's user-agent patterns. This blocks at the network level before your application processes the request.

3Switch Journey Workflows

Highest — granular, real-time control

Create a custom journey in Switch that detects OAI-SearchBot and routes it to a block action, challenge, redirect, or modified content — without touching your server configuration.

robots.txt — Block OAI-SearchBot

Add the following to your robots.txt file (at the root of your domain) to block OAI-SearchBot:

User-agent: OAI-SearchBot
Disallow: /

robots.txt — Allow with Restrictions

Alternatively, allow OAI-SearchBot on most pages while blocking specific directories:

User-agent: OAI-SearchBot
Disallow: /private/
Allow: /

OAI-SearchBot User-Agent Strings

Use these patterns to identify OAI-SearchBot in your server logs or firewall rules:

OAI-SearchBot

Frequently Asked Questions

Does blocking OAI-SearchBot affect my Google search rankings?

No. Blocking OAI-SearchBot does not affect your Google search rankings. Only blocking Googlebot impacts Google Search visibility.

Does OAI-SearchBot respect robots.txt?

Yes, OAI-SearchBot respects robots.txt directives. Adding a Disallow rule for its user-agent will prevent it from crawling blocked paths.

Can I allow OAI-SearchBot on some pages but not others?

Yes. Use robots.txt to disallow specific directories, or use Switch journey workflows for granular page-level control with conditional logic.

Go beyond robots.txt

Switch detects OAI-SearchBot in real-time and lets you build custom journey workflows — block, challenge, redirect, or serve modified content. No server changes required.

Get Started Free
OAI-SearchBot Profile Agents Directory