SEO & Webmaster
robots.txt Generator & Validator
Build, validate, and inspect robots.txt crawler directives locally, including multiple User-agent groups and Sitemap URLs.
Build rules
User-agent groups
Paste & inspect
How to use this robots.txt tool
In build mode, edit multiple User-agent groups, Allow, Disallow, and Crawl-delay rules, then add multiple Sitemaps and an optional non-standard Host. Paste an existing file in inspect mode; standard rule order, comments, and duplicate groups are retained while unknown directives and structural issues are flagged.
Semantics and compatibility boundaries
robots.txt is a crawler convention, not an authorization, authentication, or security control. An empty Disallow means no paths are disallowed; Crawl-delay is not supported by every crawler; Host is non-standard and recognized only by some search engines.
Frequently asked questions
What does an empty Disallow mean?
For example, Disallow: means that this User-agent has no disallowed path. It is not the same as blocking the whole site.
What happens with duplicate User-agent groups?
The tool does not silently merge duplicate groups. It preserves their order and reports a note because crawlers may interpret duplicates differently; check the target crawler documentation before publishing.
What does a Sitemap URL require?
A Sitemap should be a publicly reachable absolute HTTP(S) URL such as https://example.com/sitemap.xml; relative paths and other schemes are rejected.