meta-webindexer
Meta · reviewed 2026-11-05
About meta-webindexer
Indexes pages for Meta AI's web index.
Operates Facebook, Instagram and Meta AI; publishes training, ads, fetcher and web-index agents.
Usually means: An AI search index fetched this URL to decide whether it can be cited. A hit means the page is being considered, not that it was quoted.
Identity
User agent
Crawlers are matched by token, not the full browser string. Providers may add version or product text around it.
robots.txt token
Use this token in a User-agent line to allow or block the crawler.
Block this token
User-agent: meta-webindexer Disallow: /
A Disallow only stops crawlers that honor robots.txt. Classify the category before blocking; check the live file with the robots.txt checker.
Verify a hit
A matching user agent is not proof: any client can send the string. When the vendor publishes network data, compare the request against it before you trust the hit.
Match the user agent
Request UA contains meta-webindexer.
Check reverse DNS
Hostname ends with fbsv.net.
Check the IP range
Fetch the endpoint below and compare the request IP.
Reverse-DNS suffixes
fbsv.netRange endpoints
A verified hit proves the network path; it does not prove what the crawler will do with the page.
meta-webindexer: common questions
What is meta-webindexer?
meta-webindexer is a crawler from Meta, filed under AI search. Indexes pages for Meta AI's web index. Requests advertise it in the User-Agent header as meta-webindexer.
Does meta-webindexer train AI models?
No. It indexes pages for AI search results — whether a page can be cited — not to train a model.
How do I block meta-webindexer in robots.txt?
Add a group with User-agent: meta-webindexer and Disallow: / to robots.txt, then re-test the live file. robots.txt matches the token, not the full User-Agent string, and blocking this token does not block the vendor's other crawlers.
How do I verify a meta-webindexer hit?
Match the user agent, then compare the request IP with the ranges or endpoints listed on this page and check that the hostname ends with fbsv.net. A matching user agent alone is not proof — any client can send the string.
meta-webindexer vs FacebookBot: what is the difference?
FacebookBot is filed under Training, while meta-webindexer is filed under AI search. They use separate robots.txt tokens: blocking one does not block the other, and a hit from one says nothing about the other.
Testing a block? Run the live file through the robots.txt tester or check the whole AI crawler access group.
See when meta-webindexer hits your site
Page scripts never see AI crawlers because they do not run JavaScript. Install the server probe and each hit lands in the Crawlers block with its path and status.