English 中文

AI Crawler Checker

Free AI crawler checker with two modes — enter a domain to test whether GPTBot, ClaudeBot and 12 AI crawlers can actually fetch your site, or enter an IP to verify whether a visitor claiming to be an AI crawler is genuine.

Examples:

Two modes: enter a domain to read its robots.txt and fetch the homepage with 12 real AI crawler user agents; enter an IP to run reverse + forward DNS validation against official crawler domains and spot impostors.

Enter a domain or IP and hit Run Check — the crawler access table or the authenticity verdict will appear here.

How it works

  1. Site check: enter your domain — we read robots.txt and fetch your homepage with 12 real AI crawler user agents.
  2. IP verification: paste the IP address — we run reverse-then-forward DNS validation against official crawler domains.
  3. Review the access table or the authenticity verdict — fix blocks or block impostors.

Features

  • 12 AI crawlers tested: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot, cohere-ai
  • Live fetch test per crawler: allowed, blocked (403/429/503) or unreachable
  • robots.txt disallow rules parsed per crawler
  • IP verifier: reverse DNS + forward DNS cross-check flags fake crawlers
  • No AI API required — runs fully on server DNS and HTTP

FAQ

Why would an AI crawler appear in my logs but fail this test?
Log entries only show the user agent, which anyone can spoof; the live test mimics the crawler from our server. Meanwhile Cloudflare or a WAF may block the request by IP reputation — this test reveals exactly where access breaks.
Which crawlers have official IP verification?
OpenAI (openai.com), Anthropic (anthropic.com), Perplexity (perplexity.ai), Google (googlebot.com / google.com), Common Crawl (commoncrawl.org), Apple (applebot.apple.com) and Amazon (amazonbot.amazon.com) all publish reverse-DNS suffixes. Others are marked as user-agent only.
My crawler test shows blocked — what should I do?
Check three layers: robots.txt rules, your WAF / CDN rules (e.g. Cloudflare bot fight mode), and server-side user agent blocking. Allow the crawlers you want AI answers to cite your content.
Can a fake crawler pass verification?
Full verification requires reverse DNS with an official suffix AND a forward lookup resolving back to the same IP. Look-alike user agents without matching DNS records are flagged as fake.

Related tools

AI Citation CheckerCheck if Google AI Overviews cite your site for your brand keyword.GEO AuditAudit how ready your site is for AI search and generative engines.AI Fix PromptsTurn detected SEO & GEO issues into prompts ready to paste into AI coding tools.Content DifferentiationScore how unique your content is and see what to add to stand out.

Want a complete site health report?

Run a free SEO check — page, HTTPS, robots.txt, sitemap, SSL and WHOIS in one report.

Try SEO Checker