robots.txt Generator

robots.txt: tell crawlers where they may go

robots.txt is a plain-text file at the root of your domain that tells search-engine crawlers which parts of your site they may and may not request. It is the first file most crawlers fetch. This generator builds a correctly formatted file with user-agent groups, allow and disallow rules, and a sitemap reference — so you control your crawl budget without accidentally hiding pages you want indexed.

What a file looks like

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public/

User-agent: Googlebot
Disallow: /no-google/

Sitemap: https://example.com/sitemap.xml

Each group starts with one or more User-agent lines followed by its rules. The Sitemap directive is independent of any group and uses an absolute URL.

Crawling is not indexing

This is the most important and most misunderstood point. Disallow controls whether a crawler fetches a URL — it does not control whether that URL appears in search results. A disallowed page that other sites link to can still be indexed as a bare URL with no snippet. If your goal is to keep something out of the index, do the opposite of what most people expect: allow crawling and add a noindex directive, because the crawler has to read the page to see the noindex.

What robots.txt is good for

• Keeping crawlers out of internal areas (admin, carts, search-result pages)
• Conserving crawl budget on large sites by blocking low-value parameter URLs
• Advertising your sitemap location to every crawler at once
• Setting different rules for different bots via named user-agent groups

What it should never be used for

Do not use robots.txt as a security mechanism — the file is public, so listing a secret path in Disallow actually advertises it. Do not block CSS or JavaScript that Google needs to render the page. And do not rely on it to remove indexed pages; for that, use noindex or the Search Console removal tool.

Testing before you deploy

A misplaced Disallow: / blocks your entire site, so test carefully. Google Search Console includes a robots.txt report that shows the live file Googlebot sees and lets you check whether a specific URL is allowed or blocked. Validate after every change — the cost of a mistake here is your whole site disappearing from search.

Related SEO tools

• XML Sitemap Generator — build the sitemap your robots.txt should reference.
• Meta Tag Generator — set the noindex robots directive that truly keeps pages out of search.
• Canonical Tag Generator — consolidate duplicate URLs the right way.
• Schema Generator — help crawlers understand the pages you do want indexed.