Why this matters
A single stray Disallow: / can remove a site from Google. A sitemap full of redirects or error pages wastes crawl budget. This tool reads both files the way a crawler would and flags what is wrong.
Need a new file? Use the free robots.txt generator and XML sitemap generator, then read how to create and submit an XML sitemap.
Frequently asked questions
How do I test if a URL is blocked by robots.txt?
Enter your website, type the path you want to test in the optional box, choose a crawler and press Check site. The result shows whether the path is allowed or blocked and which rule matched.
Can I block AI crawlers like GPTBot and ClaudeBot?
Yes. Add a User-agent group for each AI crawler with Disallow: / in robots.txt. Use this tester to confirm the rule works, or build the file with the free robots.txt generator.
Does robots.txt remove a page from Google?
No. Disallow stops crawling, not indexing, so a blocked page can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.
What makes an XML sitemap valid?
A valid sitemap is well-formed XML with at most 50,000 URLs and 50 MB uncompressed, lists only live pages that return 200 on your own domain, and avoids redirects, errors and URLs blocked by robots.txt.
Where should my sitemap be?
Usually at /sitemap.xml on your domain. Add a Sitemap: line with its full address to robots.txt and submit it in Google Search Console so search engines find it quickly.
Read-only: we request robots.txt, your sitemaps and a few sampled URLs. Private and internal addresses are blocked.
