Skip to main content
ToolNestora

Robots.txt Generator

A robots.txt file tells crawlers which parts of a site they may visit. Pick a starting point, add the paths you want to block, and copy or download the finished file. The tool checks each path and address as you type, and nothing is sent to a server.

Generate a robots.txt file

One path per line, starting with /.

Use to re-open a path inside a blocked one.

Adds a group for GPTBot, ClaudeBot, CCBot, Google-Extended and other published crawler names. Crawlers can ignore robots.txt.

Google ignores this. Some other crawlers follow it.

Full addresses, one per line.

Upload this file to the top level of your site so it is reachable at /robots.txt. Blocking a page here stops crawling, but a blocked address can still be listed in search results. Use a noindex tag to keep a page out.

How to use it

  1. Choose a starting point

    Start from allowing everything, blocking everything, hiding common private paths, or blocking AI crawlers.

  2. Add your rules

    List paths to block and any exceptions, one per line, each starting with /. Add a rule for a single crawler or a sitemap address if you need them.

  3. Publish the file

    Copy or download robots.txt and upload it to the top level of your site, so it opens at yourdomain.com/robots.txt.

Example: keep admin pages out of the crawl

Choose “Allow all, hide admin and search pages”, add your sitemap address, and the file contains a User-agent: * group with Disallow lines for /admin/, /cart/, /checkout/ and /search, followed by a Sitemap line.

Lines you may see

A robots.txt file is made of groups of rules, each starting with the crawler they apply to.

Common robots.txt lines
LineMeaning
User-agent: *The rules below apply to every crawler that has no group of its own
Disallow: /private/Do not crawl addresses that start with /private/
Allow: /private/help/An exception inside a blocked path
Crawl-delay: 10Wait 10 seconds between requests. Google ignores it
Sitemap: https://example.com/sitemap.xmlWhere the XML sitemap is

Limits and limitations

  • robots.txt is a request, not security. Well-behaved crawlers follow it, but anything can ignore it, and it must never be used to hide private information.
  • Blocking a page from crawling does not remove it from search results. A blocked address can still be listed if other sites link to it. Use a noindex tag on a crawlable page to keep it out.
  • The AI crawler option lists 10 published user-agent names (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot and others). New crawlers appear over time, so the list is not complete.
  • The tool checks syntax, not your site. It cannot tell whether a path exists or whether you are blocking pages you want found.

Frequently asked questions

Is my information uploaded?

No. Your rules is processed by JavaScript running in your browser and is not sent to a server. The page itself still loads from a web server like any website. This site can load advertising or analytics scripts, but only after you accept them in the cookie notice. They are separate from the tools, which never pass the files or text you work with to them.

Where do I put the robots.txt file?

At the root of the host it applies to, for example https://example.com/robots.txt. A file in a subfolder is not read, and each subdomain needs its own file.

Will blocking AI crawlers stop my content being used?

It asks the named crawlers not to visit. Companies that respect robots.txt will follow it, but it does not remove content already collected and it does not bind crawlers that ignore it.

What does Disallow with nothing after it mean?

An empty Disallow line means nothing is blocked, so the crawler may visit everything. That is the “allow all” file.

More tools