Free tool by Crom Salvatera
AI Crawler Checker
See if Google, ChatGPT, Claude, and Perplexity can actually reach your site.
Enter your website and this free AI crawler checker reads your robots.txt and tests the live response your server sends to 7 search and AI crawlers. You get a clear verdict for each one, plus the fixes that matter most.
Paste a homepage or any page URL. The check takes about 5 seconds.
This AI crawler checker needs JavaScript turned on in your browser.
Quick answer
What Is An AI Crawler Checker?
An AI crawler checker is a tool that tests whether search engine and AI crawlers are allowed to reach your website. This one reads your robots.txt rules and requests your page as Googlebot, Bingbot, OAI-SearchBot, GPTBot, ClaudeBot, and PerplexityBot. It also checks the Google-Extended token. If a crawler cannot reach a page, it cannot rank it, quote it, or cite it.
Most robots.txt testers stop at the rules file. That misses half the story. A firewall, a security plugin, or a CDN setting can send a crawler a 403 Forbidden even when robots.txt says “allowed”. This AI crawler checker tests both layers, so you see what each crawler actually gets.
Layer 1
Robots.txt Rules
We apply Google’s documented matching rules to find the group and the exact Allow or Disallow line that decides each crawler.
Layer 2
Live Response
We request your page with each crawler’s user agent and record every HTTP status code, including redirects.
Control test
Normal Browser
We also request the page as a regular browser. If only the crawlers get blocked, your server is treating them differently.
The verdict
Ranked Fixes
Search crawlers come first, because blocking them costs visibility. Training crawlers are a business choice.
Why I built it
My Own Site Was Answering Crawlers With 403 Errors
In August 2026, my generative AI impressions in Google fell by about 95% in roughly a week. When I dug into the crawl data, I found 91 URLs on my own site answering with 403 Forbidden. Logged in as the owner, everything looked fine. To a crawler, those pages were locked.
I cannot prove the 403s caused the whole drop. I can tell you this: a crawler that gets a 403 cannot read the page, so it cannot rank it or cite it. After 22+ years in marketing, I know that the boring technical layer decides whether the clever content ever gets seen. So I built the AI crawler checker I wished I had that week.
The first site I ran through it was my own. It found something no plugin, setting, or SEO tool had shown me: my web host’s server turns ClaudeBot away with an HTTP 418 on every site in my hosting account, even though my robots.txt allows it. I have asked my host to fix it. That is exactly the kind of invisible block this tool exists to catch.
95%
Drop In Generative Impressions
The fall in my own Google generative AI impressions across about a week in August 2026.
91
URLs Returning 403
Pages on my own site that crawlers were told they were forbidden to read.
7
Crawlers Checked Here
The search and AI crawlers that now decide whether your business gets found, quoted, and cited.
Original research, 23 October 2026
We Checked 50 Big Australian Websites
I ran 50 of Australia’s most visited websites through this AI crawler checker: news, retail, banks, telcos, travel, property, insurance, energy, government, universities, and software. Here is what their robots.txt files tell AI crawlers today.
7 of 8
News Publishers Block GPTBot
Every major news site we checked tells OpenAI’s training crawler to stay out, except 9News, which has no robots.txt rules.
4 of 8
Also Block ChatGPT Search
The Sydney Morning Herald, The Age, news.com.au, and the AFR also block OAI-SearchBot, which opts them out of ChatGPT search answers.
0 of 32
Other Big Brands Block AI Search
None of the retail, bank, telco, insurance, energy, government, university, or software sites blocks an AI search crawler.
| Site | GPTBot | ClaudeBot | Google-Extended | OAI-SearchBot | PerplexityBot |
|---|---|---|---|---|---|
| abc.net.au | Blocked | Blocked | Blocked | Allowed | Blocked |
| news.com.au | Blocked | Blocked | Blocked | Blocked | Blocked |
| smh.com.au | Blocked | Blocked | Blocked | Blocked | Blocked |
| theage.com.au | Blocked | Blocked | Blocked | Blocked | Blocked |
| afr.com | Blocked | Blocked | Blocked | Blocked | Blocked |
| sbs.com.au | Blocked | Blocked | Blocked | Allowed | Allowed |
| 7news.com.au | Blocked | Allowed | Allowed | Allowed | Allowed |
| webjet.com.au | Blocked | Blocked | Allowed | Allowed | Allowed |
The pattern is clear. Publishers sell their words, so most block the crawlers that train AI models. Some go further and block AI search too. Businesses that sell products and services do the opposite: none of the 32 we could read blocks an AI search crawler, because being found and recommended is the point. Webjet shows the precise version: it blocks the training crawlers and explicitly allows OAI-SearchBot and PerplexityBot.
For a business that wants leads, the lesson is simple. Decide on purpose about training crawlers, and keep Googlebot, Bingbot, OAI-SearchBot, and PerplexityBot open. Then check your own site, because a firewall can still block them when your robots.txt says yes.
Method: each homepage was checked on 23 October 2026 with this AI crawler checker, using Google’s robots.txt matching rules. Googlebot and Bingbot were allowed on every site we could read.
Ten sites were left out. Nine had a robots.txt we could not read cleanly from our server. On afterpay.com, the only rule covered the bare domain’s homepage. That leaves 40 sites: 8 news publishers and 32 others.
Separately, 10 sites let our normal browser in but returned HTTP 403 to requests claiming to be Googlebot. That is usually a firewall checking that a crawler really comes from Google. Results can change at any time.
The 7 crawlers
Which Crawlers Does This Tool Check?
Not every crawler does the same job. Search crawlers decide whether you appear in search results and AI answers. Training crawlers collect content to train AI models. Blocking a search crawler costs you visibility. Blocking a training crawler is a choice about how your content gets used.
Search: Google
Googlebot
Crawls for Google Search. Google says a page must be indexed and eligible for a snippet in Search to appear as a link in AI Overviews or AI Mode, so a blocked Googlebot costs you both.
Search: Microsoft
Bingbot
Crawls for Bing search, which Microsoft Copilot uses to find web sources. My AI citations case study tracked 14,800 Copilot citations in three months.
Search: OpenAI
OAI-SearchBot
OpenAI says this crawler surfaces websites in ChatGPT search. To appear in ChatGPT search answers, OAI-SearchBot needs access.
Search: Perplexity
PerplexityBot
Perplexity says this crawler surfaces and links websites in its search results, and that it is not used to train AI foundation models.
Training: OpenAI
GPTBot
Collects content that may train OpenAI’s models. OpenAI says GPTBot and OAI-SearchBot are separate settings, so blocking GPTBot does not remove you from ChatGPT search.
Training: Anthropic
ClaudeBot
Collects content that may train Anthropic’s Claude models. Anthropic uses a separate crawler, Claude-SearchBot, for search, so blocking ClaudeBot is about training.
Training: Google
Google-Extended
A robots.txt token, not a separate crawler. Google says it controls Gemini training and grounding, and does not affect inclusion or ranking in Google Search.
Control
A Normal Browser
Our baseline request. If a browser gets in but a crawler does not, your firewall or host is filtering by user agent.
Crawlers, answered
How Do I Check If AI Bots Can Crawl My Site?
To check if AI bots can crawl your site, test two things: what your robots.txt allows, and what your server actually sends back. Enter your URL in the AI crawler checker above. In about 5 seconds it shows whether each crawler is allowed, blocked by a rule, or blocked by your server, with the exact rule or HTTP code behind each result.
For a second opinion, open your robots.txt at yourdomain.com/robots.txt and look for groups naming each crawler. Then check your server or CDN logs for each crawler’s visits and their status codes. Google Search Console’s URL Inspection tool shows how Googlebot sees a single page.
Why Does My Website Return 403 To Bots?
A website returns 403 Forbidden to bots when something in front of the page decides that visitor is not welcome. Common causes are CDN bot settings such as Cloudflare’s AI bot blocking, security plugins with bot or rate limiting rules, host firewalls, and server rules that block user agents by name.
These blocks are easy to miss, because you browse your own site as a normal visitor and everything loads. The AI crawler checker compares each crawler against a normal browser request, so you can see whether the block targets specific crawlers or all automated traffic.
Use it like a pro
How To Fix A Blocked Crawler In 5 Steps
-
Run The Check On Your Key Pages
Test your homepage, a service page, and a blog post. Rules and firewalls can treat each one differently.
-
Fix Search Crawlers First
Googlebot, Bingbot, OAI-SearchBot, and PerplexityBot decide whether you get found and cited. Start there.
-
Clean Up Your Robots.txt
Remove Disallow rules that block search crawlers by mistake, and keep one clear group per crawler.
-
Check Your Firewall, CDN, And Plugins
If robots.txt allows a crawler but the server returns 403, the block lives in your CDN, host, or security settings.
-
Run The Check Again
Use “Check again” after each change. Then watch Search Console and Bing Webmaster Tools over the next few weeks.
Should I Block GPTBot Or ClaudeBot?
Block GPTBot or ClaudeBot only if you have decided you do not want your content used to train AI models. Both are training crawlers. OpenAI and Anthropic each run separate search crawlers, OAI-SearchBot and Claude-SearchBot, so blocking the training crawlers does not, on its own, remove you from ChatGPT search or Claude’s search answers.
The costly mistake is blocking the wrong one. A blanket rule that blocks every AI crawler also blocks the search crawlers that send you visitors and citations. The AI crawler checker splits your results into search and training, so you can make that call on purpose.
Does Robots.txt Stop AI From Using My Content?
Robots.txt tells well-behaved crawlers which pages they may fetch. OpenAI, Anthropic, Google, Microsoft, and Perplexity all publish robots.txt controls for their crawlers. It is a request, not a lock, and it only affects future crawling.
Some fetches also work differently. OpenAI says ChatGPT-User visits pages when a person asks, and that robots.txt rules may not apply to those visits. If you need a page kept private, put it behind a login. If you want a page found, make sure robots.txt and your server both let the search crawlers in.
Who it is for
Who Is This AI Crawler Checker For?
Business Owners
Find out in seconds whether Google, ChatGPT, and Perplexity can even see your site before you spend another dollar on content.
SEO And Content Teams
Rule out crawl access before you diagnose a ranking or AI visibility drop. It is the first thing I check.
Developers And Hosts
Confirm that a firewall, CDN, or plugin change has not quietly started blocking the crawlers that bring in traffic.
Not A Log Analyser
This tool simulates each crawler from our server. Your server logs remain the record of real crawler visits.
Go further
Access Is Step One. Getting Cited Is The Goal.
Letting the crawlers in is the floor. An AI Marketing Audit shows what else stands between your business and being found in Google and cited by AI answers, and what to fix first. Ongoing work runs through my SEO, AEO, and GEO retainer.
FAQ
Frequently Asked Questions About AI Crawler Checkers
Is This The Same As A Real Crawler Visit?
Not exactly. The AI crawler checker sends each crawler’s published user agent from our server, not from Google’s or OpenAI’s networks. Some firewalls verify crawlers by IP address, so a real crawler can get a different result. Treat a block here as a strong signal, then confirm it in your server logs.
Does Google-Extended Affect My Google Rankings?
No. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. It controls whether Google can use your content for Gemini training and grounding.
What Happens If My Robots.txt Returns An Error?
Google treats a missing robots.txt (a 404) as no restrictions. A server error is different. Google says it stops crawling the site for the first 12 hours while it keeps trying to fetch the file, so a broken robots.txt can pause crawling.
Why Does One Crawler Show A Different Code?
Your server, CDN, or a security plugin is likely matching that crawler’s user agent. Codes such as 403, 406, 418, or 429 usually mean a rule is turning it away, even when robots.txt allows it.
How Often Should I Check?
Check after any change to your host, CDN, security plugin, or robots.txt, and at least once a month. Crawler access breaks quietly, and you usually notice only when traffic drops.
Do You Store The Sites I Check?
Each result is cached for 15 minutes so repeat checks stay fast and free. To stop abuse, we also keep an anonymous count of checks per connection for one hour. We do not ask for your name or email to run the check.
What to do next
Turn Crawler Access Into Leads
Run the AI crawler checker on your homepage and your top money pages. Fix any search crawler that is blocked, then decide on purpose what to do with the training crawlers.
Once the crawlers can get in, the work shifts to being worth citing. Read how I approach answer engine optimisation and generative engine optimisation, and see the AI citations case study.
More free tools
About Crom Salvatera
Crom Salvatera is a Sydney-based AI marketing consultant and Head of SEO, AEO and GEO with 22+ years in marketing and 14 in high-level digital. He has managed and optimised 500+ ad accounts, has helped generate $650M+ in revenue for employers and clients, and created the TLC Method (Tech, Links, Content). His brand experience includes LEGO, Hasbro, JB Hi-Fi, and Crimson Education. Connect with him on LinkedIn.
References
- Google Search Central: How Google interprets the robots.txt specification
- Google Search Central: Google’s common crawlers (Google-Extended)
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
- Google Search Central: AI features and your website