Meta-WebIndexer
Meta's web indexer for improving Meta AI search and knowledge.
What is Meta-WebIndexer?
Meta-WebIndexer is Meta's dedicated web indexing crawler for improving Meta AI's search capabilities and knowledge base. This is separate from Meta's social crawlers (like facebookexternalhit) which generate link previews.
Meta has been expanding its AI capabilities rapidly with Llama models and Meta AI integration across Facebook, Instagram, and WhatsApp. Meta-WebIndexer builds the knowledge base that powers Meta AI's ability to answer questions and provide information.
This is a relatively new crawler reflecting Meta's push into AI-native search within its social platforms. Content indexed by Meta-WebIndexer may surface when users interact with Meta AI across any of Meta's platforms, reaching billions of users.
User-Agent Strings
These are the known user-agent patterns used by Meta-WebIndexer. Use them to identify this crawler in your server logs or configure robots.txt rules.
robots.txt example:
User-agent: Meta-WebIndexer Disallow: /private/ Allow: /
How to Manage Meta-WebIndexer
Allow to be accessible in Meta AI across Facebook, Instagram, and WhatsApp.
Separate from social crawlers — manage AI indexing independently.
Use Switch to differentiate Meta-WebIndexer from facebookexternalhit.
Meta AI reaches billions — high visibility potential.
Start managing Meta-WebIndexer today
Switch detects, tracks, and lets you build custom journeys for Meta-WebIndexer and 35+ other AI agents and crawlers. Set up in five minutes.
Get Started FreeRelated Agents
Meta Crawler
Social CrawlersMeta
Meta's suite of crawlers for Facebook, Instagram, and WhatsApp link previews.
AI2Bot
Commercial CrawlersAllen AI
Allen Institute for AI's research crawler for academic AI development.
Amazonbot
Commercial CrawlersAmazon
Amazon's web crawler powering Alexa, Amazon search, and AI services.
Applebot-Extended
Commercial CrawlersApple
Apple's AI training token controlling how Applebot data is used for Apple Intelligence.
Bytespider
Commercial CrawlersByteDance
ByteDance's web crawler for TikTok AI and LLM training data.
CCBot
Commercial CrawlersCommon Crawl
Common Crawl's open-source web archive used by multiple AI companies for training.