Robots.txt Generator

A plain-text file at the root of a site telling crawlers which parts of it they may fetch. It is a request, not a barrier: well-behaved crawlers read it and honour it, and anything else ignores it. The format was standardised as RFC 9309 in 2022, after nearly thirty years as a convention.

At the root of the host, at /robots.txt — https://example.com/robots.txt and nowhere else. A file in a subdirectory is never read. It is also per-host, per-scheme and per-port: https://example.com and https://blog.example.com need their own, and so, strictly, do the http and https versions of the same host.

It opens the fallback group — the rules for any crawler that has no group of its own. The important half is what it does not mean: a crawler with its own group ignores the * group completely. Give Googlebot a group and it stops reading the * rules, even the ones you meant to apply to everything.

It asks the matching crawlers not to fetch any URL on the site. It is the correct line for a staging site and the single most damaging line a live site's robots.txt can contain. Note the contrast with an empty `Disallow:`, which explicitly disallows nothing — one character apart, opposite meanings.

Use Allow for the exception, in the same group: `Disallow: /private/` followed by `Allow: /private/public/`. Google and Bing resolve the two by picking the longest matching rule, so the more specific Allow wins for URLs under /private/public/. Where two rules are the same length, Google breaks the tie in favour of Allow.

It can ask Googlebot not to crawl, and Googlebot honours that. What it cannot do is keep the URL out of the search results — Google may still index a blocked URL it finds linked elsewhere, listing it without a description because it was never allowed to read the page.

No, and using it that way usually backfires. To keep a page out of the index you need a `noindex` meta tag or an `X-Robots-Tag` header — and Google has to crawl the page to see either. Blocking the page in robots.txt prevents that, so a page blocked and tagged noindex can stay indexed indefinitely. Let it be crawled, serve noindex, and remove the block once it has dropped out.

No. It is a public file — anyone can read yours right now — and it has no enforcement of any kind. Listing `Disallow: /admin/` tells every visitor exactly where your admin panel is while doing nothing to stop them opening it. Anything that must not be reached needs authentication and authorisation on the server. If a path is sensitive, keep it out of robots.txt entirely.

Disallow asks crawlers not to fetch URLs matching a path; Allow carves an exception out of a Disallow. Allow was not in the original 1994 convention — it arrived later and is now part of RFC 9309 and supported by every major crawler — which is why some very old parsers understand only Disallow.

One line: `Sitemap: https://example.com/sitemap.xml`. It must be an absolute URL including the scheme; a relative path is ignored. Sitemap lines are non-group directives, so they sit outside every User-agent block and apply to all crawlers regardless of which group they read. You can list several.

Yes. Anything after a # is a comment, whether it starts the line or follows a directive. Comments are worth using — a robots.txt is edited rarely and by whoever happens to be on call, and a line saying why a path is blocked saves someone unblocking it by mistake.

No. Honouring it is voluntary. The major search engines do, and the well-known AI crawlers publish tokens precisely so they can be blocked. Scrapers, harvesters and anything acting in bad faith do not, and nothing in the file can make them — which is the same reason it is not a security measure.

A request to wait a given number of seconds between fetches, intended to relieve load on a small server. It is not in RFC 9309 and support is uneven.

No. Google has never honoured Crawl-delay and says so explicitly; the line is read and discarded. Bing, Yandex and several SEO crawlers do honour it. For Google, crawl rate is adjusted in Search Console — and if crawling is genuinely overloading your server, returning 429 or 503 is the signal every crawler understands.

Yes, and it is the normal way to give different crawlers different rules. The thing to remember is that a crawler obeys exactly one group — the most specific one matching its name — and rules never accumulate across groups. Anything from the * group that should also apply to a named crawler has to be repeated inside that crawler's own group.

Half of it. Directive names and user-agent tokens are case-insensitive, so `Disallow` and `disallow` are the same and `Googlebot` matches `googlebot`. Paths are case-sensitive, because URLs are — `/Admin/` and `/admin/` are different rules and blocking one does not block the other.

Yes, with the major crawlers and under RFC 9309. `*` matches any run of characters and `$` anchors the end of the URL, so `Disallow: /*?sort=` blocks any URL containing that parameter and `Disallow: /*.pdf$` blocks URLs ending in .pdf. A trailing `*` on a folder is redundant, though — matching is already a prefix match, so `/admin/*` and `/admin/` are identical.

By length. Google and Bing compare every rule in the chosen group against the URL and apply the one with the longest matching path, whether it is an Allow or a Disallow. Order in the file is irrelevant. When two matching rules are the same length, Google prefers Allow.

robots.txt controls crawling — whether a URL may be fetched at all — and lives in one file for the whole site. The meta robots tag and the X-Robots-Tag header control indexing of one page, and are only seen if that page is crawled. Use robots.txt to keep crawlers out of areas that waste crawl budget, and noindex to keep a page out of results. Using both on the same URL is self-defeating.

You can ask them not to crawl, and the major operators publish tokens for exactly that — GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, CCBot for Common Crawl. Google-Extended is a special case: it is not a crawler and fetches nothing, but disallowing it opts your content out of Gemini training and grounding without affecting Search. As with every other crawler, compliance is voluntary.

A 404 means no restrictions, and crawlers proceed as if the file were empty — which is fine for most sites. A 5xx is different: Google treats a persistent server error as a signal to stop crawling the site entirely, on the assumption that the file exists but cannot be read. A broken robots.txt endpoint is worse than no robots.txt.

Google reads the first 500 KiB and ignores everything after it. That is a very large robots.txt, but a file generated from a database of URLs can reach it — and the failure is silent, because the truncated rules simply stop applying. Prefer patterns over long lists.

No. `Noindex:` was never documented by Google and stopped working in September 2019 — use a noindex meta tag instead. `Host:` was a Yandex extension and is deprecated even there; state a preferred host with a canonical link or a redirect. Both are ignored by everything else, so a file relying on them is not doing what it appears to.

Yes. The builder, the raw editor, the validation, copy and download are free, with no account and no limit.

No. Everything runs locally in your browser — the file is generated, validated and downloaded on your own machine, and nothing you type is sent to our server. There is no account and nothing is stored.

Yes, as robots.txt with the correct name, ready to upload to the root of your site. You can also copy it to the clipboard, or switch to the raw editor and adjust it by hand first.

Yes — the Raw / edit tab hands you the text. Nothing is rewritten while you type, comments are preserved, and validation runs against exactly what is in the box. Switching back to the builder reads your text and rebuilds the form from it; that normalises spacing and hoists any trailing comment to the top, which the tab says before you do it.

Not directly — there is no ranking signal in it. Indirectly it matters a great deal, because a page that cannot be crawled cannot be ranked on its content, and a badly aimed Disallow can remove a whole section from consideration. Blocking crawl-wasting URLs such as faceted search can help by concentrating crawling on pages worth indexing.