ROBOTS_BLOCKS_AI_CRAWLER
robots.txt blocks a live AI crawler
Synviber can fix this automatically — no connected domain requiredWhat this check looks at
Whether robots.txt disallows any of the AI crawlers that fetch and cite content live — blocking a training-only crawler (GPTBot, ClaudeBot, CCBot, and similar) does not trigger this check, since that's a legitimate, product-endorsed choice.
Why AI answer engines care
Blocking a crawler that fetches and cites content live (as opposed to one that only collects model-training data) directly prevents an AI answer engine from citing the site.
Does Synviber fix it for me?
Yes. Synviber's GEO scan can generate and publish a fix for this without a connected domain — it writes the suggestion to your Knowledge Hub, or publishes it live immediately if the domain is connected.
How to fix it manually
High severity
Scanner message: "robots.txt blocks AI crawlers that fetch and cite content live: <comma-separated bot list>"
Add Allow rules for these user agents in robots.txt so AI answer engines can fetch and cite your content. (Blocking training-only crawlers like GPTBot/ClaudeBot/CCBot does not affect this score — that's a legitimate choice enforced by the platform's Gateway pillar instead.)