Robots.txt Generator
Build a clean, standards-compliant robots.txt file in a few clicks. Guide search engine crawling the right way.
Settings
Your robots.txt
Save the file in your domain root: https://yoursite.com/robots.txt
What is robots.txt?
Robots.txt is a text file in the root of your site that tells search engine crawlers which parts of the site they may index and which they may not. A correctly written robots.txt steers your crawl budget toward important pages and keeps things like admin panels and shopping carts out of search results.
Be careful not to accidentally block important pages, though – a faulty robots.txt can drop your entire site from Google. Don't use robots.txt to hide sensitive content either, since the file is public. For that, use a noindex tag or password protection instead.
- Block admin, cart and search result pages from indexing.
- Always add a sitemap URL so Google finds all your pages.
- Test the file with Google Search Console's robots.txt tester.
Want to make sure your technical SEO is in order? Explore our technical SEO service or request a free audit.
