What robots.txt does
The robots.txt file sits at the root of your site and tells crawlers which parts they may visit. It controls crawling, not indexing, and it is a request that well-behaved crawlers follow. A mistake can hide your whole site from search engines, so generate it carefully and test it afterwards.
Blocking AI crawlers
Several AI companies publish crawler names you can disallow, including GPTBot, ClaudeBot, Google-Extended and CCBot. Blocking them asks those companies not to use your pages for training. It does not affect your search rankings.
Frequently asked questions
Will robots.txt hide a page from Google?
Not reliably. A blocked page can still be listed if other sites link to it. Use a noindex tag or a password for pages that must stay out of search.
Do I need a robots.txt file?
Not strictly, but it is a good place to list your sitemap and keep crawlers out of search results pages, carts and admin areas.
How do I test the file?
Use the robots.txt and sitemap tester to check whether specific addresses are allowed for Googlebot, Bingbot and AI crawlers.
Read-only: we request public pages the way a visitor would. Private and internal addresses are blocked, and nothing you enter is stored.
Want this running on every page, every day?
UrKavach monitors security, uptime, speed and SEO for every website you manage.