Choose a starting policy, inspect the snippet, and merge it with your existing robots.txt.
robots.txt expresses preferences. It is not an access-control mechanism. Some user-triggered fetchers may ignore these rules. “All listed” covers this tool’s documented OpenAI, Anthropic and Perplexity records; it cannot cover every AI agent.
Merge carefully
Keep existing disallowed paths and sitemap declarations. A new specific group can replace the wildcard policy for that crawler, so copy any applicable path restrictions into the specific group. Review the completed file before publishing.
No. This generator covers documented OpenAI, Anthropic and Perplexity tokens. Robots.txt expresses preferences and does not enforce access control. Review the snippet and preserve existing restrictions.