Loading page
About this tool
Create standard robots.txt files for search engines with interactive forms to configure crawling rules, delays, and sitemaps.
The Robots.txt Generator builds a valid Robots Exclusion Protocol file from a form: a global User-agent: * group that either allows everything (an empty Disallow:) or blocks everything (Disallow: /), a list of disallowed paths, per-crawler Allow and Disallow rules grouped by user agent, an optional Crawl-delay, and a Sitemap line. The preview updates as you type and you can copy it or download it as robots.txt ready for your site root. It is for site owners who know which directories should stay out of search results but not the exact syntax that expresses it.
Open Robots.txt Generator on AltFTool — it loads instantly in your browser.
In the Crawler Configuration Form set Global Bot Access to "Allow All Bots (*)" or "Block All Bots (*)", pick a Crawl Delay of 2, 5 or 10 seconds, and paste a Sitemap Path such as https://example.com/sitemap.xml.
Add each path under Global Disallowed Directories, then use "Add Bot-Specific Directives" to pair Googlebot, Bingbot, Baiduspider, Yandex, Yahoo Slurp or Alexa ia_archiver with Disallow or Allow and a path.
"File Preview (robots.txt)" rebuilds on every change so you see the literal text that will ship; Copy puts it on the clipboard and Download saves it as robots.txt for your site root.
Rules for the same crawler are collected into one User-agent block rather than scattered, which is how parsers expect directives to be scoped.
The output rebuilds on every change, so you see the literal text that will ship instead of guessing what the form produced.
The Sitemap line is emitted independently of the user-agent groups, and an optional Crawl-delay sits with the global group.
At the root of the host it applies to — https://example.com/robots.txt — and nowhere else. A file in a subdirectory is ignored, and each subdomain and each scheme needs its own, so www and non-www are separate files as far as crawlers are concerned.
Use User-agent: * followed by Disallow: / , which is what the block-everything option generates. An empty Disallow: line means the opposite — it allows everything — so the difference between blocking and allowing your entire site is the single slash.
No. Crawl-delay was never part of the original Robots Exclusion Protocol and Google's crawlers ignore it; some other crawlers, including Bing and Yandex, do honour it. Use Search Console's crawl rate settings if you need Googlebot to slow down.
Not reliably. Disallow stops crawling, not indexing — a blocked URL linked from elsewhere can still appear in results without a description. To keep a page out of the index, allow it to be crawled and serve a noindex meta tag or X-Robots-Tag header instead, and remember robots.txt is public and is not an access control.