About the Robots.txt Generator
The Robots.txt Generator builds the small text file that tells crawlers which parts of your site they may fetch. Start from a preset, edit the rule groups, add your sitemap, and download a ready-to-upload robots.txt. A built-in tester shows whether a given path is allowed or blocked for a given crawler, and which rule decided it.
The format follows RFC 9309, the Robots Exclusion Protocol standard published in 2022, which Google, Bing and other major search engines follow. Groups start with one or more User-agent lines, then list Disallow and Allow paths, and Sitemap lines can go anywhere in the file.
Keep in mind what robots.txt is for. It manages crawling, not indexing or security: a blocked page can still appear in results if other sites link to it, and anyone can read the file. Well-behaved crawlers respect it, but nothing forces them to. The tool runs entirely in your browser.
How to use it
- Pick a starting pointAllow everything, Block everything, WordPress or Online shop. Each one fills in the rule groups for you.
- Edit the groupsSet the crawlers for each group (* means all), then list the paths to block and the paths to allow, one per line. Add more groups for specific crawlers.
- Add your sitemapEnter the full sitemap URL, or several, one per line. Tick the AI crawler option if you want to block those bots.
- Test a few pathsType a path such as /cart and a crawler such as Googlebot. The verdict names the rule that matched.
- Upload the fileDownload robots.txt and put it at the root of your site, so it opens at https://yoursite.com/robots.txt.
Key features
Four presets
Open site, fully blocked site, WordPress (admin blocked except admin-ajax.php, search pages blocked) and an online shop (cart, checkout, account and sort or filter URLs blocked).
Any number of groups
Give Googlebot, Bingbot or any other crawler its own rules, next to the rules for everyone else.
Standard path matching
The tester uses the RFC 9309 rules: * matches any characters, $ marks the end of the URL, the longest matching rule wins, and Allow wins a tie.
AI crawler block
One switch adds a group for GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, cohere-ai and PerplexityBot.
Checks while you type
Warnings for a file that blocks the whole site, for Crawl-delay, for paths that do not start with / and for sitemap URLs that are not full addresses.
Private by design
Nothing is uploaded or fetched. The file is built and tested on your device.
Examples
The WordPress preset
User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /search/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml| Path | Result |
|---|---|
/wp-admin/admin-ajax.php | Allowed by "Allow: /wp-admin/admin-ajax.php" |
/wp-admin/options.php | Blocked by "Disallow: /wp-admin/" |
/?s=mugs | Blocked by "Disallow: /?s=" |
/blog/hello | Allowed (no rule matches) |
The admin-ajax.php path is allowed even though it sits inside the blocked folder, because its rule is longer and therefore more specific.
Prefixes, wildcards and the end of a URL
Rules match from the start of the path, and they match prefixes. In the Online shop preset:
| Path | Result |
|---|---|
/cart | Blocked by "Disallow: /cart" |
/cartoons | Also blocked by "Disallow: /cart" |
/mugs?sort=price | Blocked by "Disallow: /*?sort=" |
/mugs?color=blue&sort=price | Blocked by "Disallow: /*&sort=" |
That is why the preset blocks both ?sort= and &sort=: the first only matches when sort is the first parameter. To block a single URL and nothing that starts like it, end the rule with $. With Disallow: /*.pdf$, the path /files/menu.pdf is blocked, but /files/menu.pdf?v=2 is not.
A group for one crawler replaces the * group
Add a second group for Googlebot with Disallow: /drafts/ next to the shop rules. Now /cart is allowed for Googlebot, because a crawler follows only the group that names it and ignores the * group. Bingbot, which has no group of its own, still follows the * rules. If you give a crawler its own group, copy the general rules into it.
Tips
- Keep it short. Block only what crawlers should never fetch, such as carts, internal search results and endless filter combinations.
- To keep a page out of search results, use a noindex meta tag and let the page be crawled. A robots.txt block hides the tag from search engines.
- Paths are case-sensitive: Disallow: /Private/ does not block /private/.
- Each host needs its own file. https://shop.example.com/robots.txt does not apply to https://example.com.
- After uploading, check the file in Google Search Console's robots.txt report to see the version Google fetched and any parsing problems.
Common mistakes and limits
- Blocking CSS and JavaScriptSearch engines render pages much like a browser. If they cannot fetch your styles and scripts, they may misjudge the page, especially on phones.
- Using robots.txt to hide private pagesThe file is public, so listing secret paths advertises them. Protect private areas with a login, and keep them out of results with noindex.
- Expecting every bot to obeyrobots.txt is a request, not an access control. Search engines and most AI companies follow it, but some crawlers ignore it.
- Relying on Crawl-delay for GoogleGoogle ignores Crawl-delay. Bing supports it, and other crawlers vary. Google adjusts its crawl rate to how fast your server responds.
- Leaving "Disallow: /" from a staging siteA staging file copied to the live site blocks search engines from everything. Check robots.txt after every launch or migration.
