Robots.txt Generator
Configure user-agent groups, allow and disallow rules, crawl delay and your sitemap URL, then copy or download a valid robots.txt.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /*?s= Allow: / Sitemap: https://example.com/sitemap.xml
100% private
This tool runs entirely in your browser. Nothing you type is uploaded, logged or stored.
Crawl Budget, Crawler Directives and robots.txt Rules
robots.txt is a plain-text file at your domain root that tells crawlers which paths they may request. It is grouped by user-agent, with Disallow and Allow directives beneath each group. Rules are prefix matches, `*` acts as a wildcard and `$` anchors the end of a URL, so `Disallow: /*.pdf$` blocks PDF files anywhere on the site. The most specific matching rule wins, which is how an Allow can carve an exception out of a broader Disallow.
The single most important thing to understand is what robots.txt does not do. Blocking a URL prevents crawling, not indexing — a blocked page can still appear in results if other sites link to it, just without a description. To keep a page out of the index, allow crawling and serve a noindex meta tag or header. Blocking it in robots.txt actively prevents the crawler from ever seeing that directive.
Use it instead to protect crawl budget on large sites: block faceted-search parameter URLs, internal search results, infinite calendar pages, cart and checkout paths, and staging subdirectories. Never block CSS or JavaScript that the page needs to render, because search engines evaluate the rendered layout. Always list your sitemap URL, keep the file under 500 KB, and test changes before shipping — one stray `Disallow: /` deindexes an entire site.
Frequently asked questions
At the root of the domain, for example https://example.com/robots.txt. Crawlers only look there; a file in a subdirectory is ignored.