← All text tools

Robots.txt tester

Fetch or paste a robots.txt and test whether Googlebot and other crawlers may visit a URL.

  • Free
  • No sign-up
  • Works on any device

About Robots.txt tester

Fetch a site's robots.txt or paste your own, then test whether a crawler like Googlebot may access a given URL. Rules are matched by user-agent and the longest matching path wins. Remember that robots.txt blocks crawling, not indexing, so blocked URLs can still appear in search.

How to use Robots.txt tester

  1. Type what you want to look up above.
  2. The answer appears instantly, explained in plain words.
  3. Copy what you need.

Why use this tool?

Also known as

Frequently asked questions

Does robots.txt hide a page from Google?

Not reliably. It stops crawling, but a blocked URL can still be indexed if linked elsewhere. To keep a page out of results, use a noindex tag or password protection.

Is Robots.txt tester free?

Yes. Robots.txt tester on Toolipedia is free to use, with no sign-up and no watermark.

Is it safe to use?

Yes. Everything is worked out in your browser, and nothing you enter is sent to our server.

Does it work on my phone?

Yes. It works in any modern browser on Android and iPhone, as well as on tablets and computers. There's nothing to install.

Do I need an account?

No. Just open the page and use the tool. There's no sign-up and no email needed.

Related tools