Robots.txt generator
Choose which folders to keep crawlers out of, add your sitemap, and decide how AI crawlers are treated. Copy the result to /robots.txt at your site's root.
Your robots.txt
Save at the root of your site, for example yourwebsite.com/robots.txt.
User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://yourwebsite.com/sitemap.xml
Questions
About The robots.txt generator
New to SEO? Read SEO Vs AEO Vs GEO: What They Mean And How To Do Each.
Does robots.txt hide pages from Google?
No. It stops crawling, but a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of results.
Should I block AI crawlers?
Blocking AI search bots keeps you out of ChatGPT, Claude and Perplexity answers. Many businesses allow search bots and block only training bots.
Technical
All 22 Tools →Related Tools
XML sitemap generatorCrawl a site (up to 100 pages) and get a sitemap.xml.Live URLRedirect and status checkerFollow a URL and see every redirect hop and status code.Live URLIndexability checkerCan Google index this page? Status, canonical, noindex and robots.txt.Live URLHeading structure checkerSee a page's H1–H3 outline and fix missing or skipped headings.Live URLImage alt text checkerList every image on a page and whether it has alt text.Live URLPage speed checkerMobile and desktop speed scores with Core Web Vitals.Live URLHreflang tag generatorLanguage and region tags for multilingual or multi-country sites.In Your BrowserSocial preview checkerSee how a link looks when shared, and which social tags are missing.Live URL
Free · Up To 50 Pages
Check Your Whole Site, Free
Paste your homepage and CiteZap checks every page and writes the keywords, titles, descriptions, schema and fixes.