Robots.txt Generator

Free robots.txt generator — build a correct robots.txt with allow, disallow, sitemap and crawl-delay rules in seconds.

Rules

Sitemap URL (optional)

User-agent (comma-separated for multiple)

Disallow paths (one per line)

Allow paths (one per line, optional)

Crawl-delay seconds (optional)

robots.txt Output

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

How it works

Pick a preset: allow everything, block everything, WordPress or block AI crawlers.
Fine-tune user-agent groups, disallow and allow paths, then add your sitemap URL.
Copy the generated robots.txt or download it as a file.

Features

WordPress, allow-all, block-all and block-AI presets
Unlimited user-agent groups with allow and disallow rules
Sitemap and crawl-delay directives supported
One-click copy and download with live preview

FAQ

What is robots.txt?

robots.txt is a plain text file at the root of your site that tells crawlers which paths they may or may not access.

Where do I put the generated file?

Upload it to the root of your domain so it is reachable at https://yourdomain.com/robots.txt.

Does robots.txt remove pages from Google?

No. robots.txt controls crawling, not indexing. To remove a page from search results, use a noindex meta tag or the Removals tool in Search Console.

Should I block AI crawlers?

It depends on your goals. Blocking GPTBot, CCBot and similar agents stops AI training crawls but also removes you from AI answers. Use the preset and adjust as needed.

Website Analyzer Coming soonFull on-page analysis: meta, headings, images and more.

SEO CheckerInstant SEO report: meta tags, robots.txt, sitemap, SSL and WHOIS.Meta Tag AnalyzerInspect title, description, Open Graph and robots meta tags.

Want a complete site health report?

Run a free SEO check — page, HTTPS, robots.txt, sitemap, SSL and WHOIS in one report.

Try SEO Checker