Frequently asked questions
Does this tool monitor my robots.txt over time?
No. Each comparison is a single bounded fetch and a stateless diff run when you submit the form. No schedule, alert or history is kept; nothing is saved after the response is returned.
What counts as a change?
Rules are compared per user-agent group as exact Allow/Disallow lines, plus declared Sitemap entries. Added and removed items are listed separately; nothing is inferred about actual crawler behavior.