UtiliHub
Español

robots.txt generator

Build a robots.txt with user-agent groups, allow and disallow rules, sitemap, and a local path check.

Loading tool…

Recommended next step

Related developer tools

After using robots.txt generator, continue with Cron expression generator.

Open next

Continue this path

Tools selected to complete the same task.

4 steps

Tool guide

robots.txt generator: how it works

Practical information to understand the result and use this tool correctly.

About robots.txt generator

A robots.txt file tells cooperating crawlers which paths they may fetch. Each group starts with User-agent, then one or more Allow or Disallow lines. A blank line separates groups. Sitemap is a full URL and sits outside the groups.

The path check uses a longest-match rule: * matches any text and $ anchors the end. If two rules match the same length, Allow wins. A path with no matching rule is allowed. This is a preview, not a live crawl of your site.

Disallow: / blocks the whole host for that agent. Disallow: /admin blocks /admin and /admin/users. Googlebot ignores Crawl-delay; some other bots still read it. Blocking a bot here does not remove pages already indexed.

Nothing is uploaded. Copy the file or download robots.txt and place it at the site root, for example https://example.com/robots.txt.

How to use it

  1. 1Pick a preset or add a user-agent group.
  2. 2Add Allow and Disallow paths that start with /.
  3. 3Optional: set crawl-delay, a comment, and an https sitemap URL.
  4. 4Check a sample path, then copy or download robots.txt.

Frequently asked questions

How do I block the whole site?
Use User-agent: * and Disallow: /. That group matches every agent that does not have its own group. A path check of /blog returns blocked.
How do I allow the site but hide /admin?
Allow: / and Disallow: /admin. Longest match wins, so /admin/users is blocked and /blog stays allowed. Paths must start with /.
Does Crawl-delay work on Google?
No. Googlebot ignores Crawl-delay. A value of 10 is written for other bots. Leave it empty if you do not need it. Negative or non-numeric values are rejected.
Where does the Sitemap line go?
After the groups, as Sitemap: https://example.com/sitemap.xml. It must be an http or https URL. An empty sitemap is omitted.
Is the file uploaded?
No. Rules and the sitemap URL stay in the browser. Download robots.txt and upload it yourself to the site root.