Four Ways to Generate a robots.txt File
This free robots.txt generator, and a robots.txt generator online you can use to generate robots.txt file output from any device, covers four common situations. This seo friendly robots.txt preset is the sensible default for most sites, disallowing typical non-content paths (admin, API, private areas) while leaving everything else crawlable, essentially a robots.txt disallow generator for the paths that matter most. Robots.txt allow all crawlers mode produces the simplest possible file, permitting every crawler to access everything, useful for a brand-new site with nothing to hide from search engines yet. Robots.txt block all crawlers mode does the opposite, blocking every crawler from the entire site, typically used on staging or pre-launch environments that shouldn't be indexed at all. Custom robots.txt rules generator mode gives full control over user-agents, allow paths, disallow paths, specific-bot blocking, and crawl delay, for sites with more particular crawling requirements than the SEO-friendly defaults cover.
How to Use This robots.txt Generator
Pick a preset from the toolbar. For SEO-friendly or Custom, fill in User-agents (defaults to * for all crawlers), Allow paths and Disallow paths (one per line), Block specific bots for crawlers you want excluded entirely regardless of the main rules, and an optional Crawl-delay in seconds. Add a Sitemap URL if you want to declare your sitemap's location directly in robots.txt, this robots.txt sitemap directive line helps crawlers discover your sitemap without needing it submitted separately everywhere. Results update automatically as you edit any field, then copy the output or download it directly as robots.txt.
What Belongs in a robots.txt File
A robots.txt file uses a small, well-defined set of directives, each on its own line, grouped into blocks that each start with a User-agent line. This robots.txt generator online produces syntactically correct output for every directive it supports, so you don't need to memorize the exact syntax or worry about a typo silently breaking a rule.
| Directive | Purpose | Example |
|---|---|---|
| User-agent | Which crawler the following rules apply to | User-agent: * |
| Allow | Explicitly permits crawling a path (useful to carve out an exception inside a disallowed directory) | Allow: /public/ |
| Disallow | Blocks crawling of a path and everything under it | Disallow: /admin/ |
| Crawl-delay | Requests a minimum number of seconds between requests (not honored by all crawlers, see FAQ) | Crawl-delay: 5 |
| Sitemap | Points crawlers to your sitemap's location, can appear anywhere in the file | Sitemap: https://example.com/sitemap.xml |
Blocking Specific Bots Without Blocking Search Engines
The Block specific bots field generates a separate User-agent block with Disallow: / for each bot you list, placed before your main rules, so a named bot (an SEO-scraper crawler, an aggressive AI-training crawler, or any other bot you don't want indexing your content) is blocked entirely while your main User-agent: * rules, and Googlebot, Bingbot, and other legitimate search engines, are completely unaffected. This is the correct way to target one specific crawler without touching the rules everyone else follows.