What Is ClaudeBot? Anthropic's Web Crawler Explained
By the Chatgbot Team · Published August 24, 2026
ClaudeBot is the web crawler operated by Anthropic, the company behind the Claude AI assistant. It automatically visits public web pages to gather text data, and site owners can allow or block it through their robots.txt file.
If you run a website, you have probably seen ClaudeBot in your server logs and wondered what it is doing and whether you should let it through. This guide explains the ClaudeBot crawler in plain terms, including its user agent string, how it behaves, and the exact steps to allow or block it.
It also clears up a common mix-up: the Claude bot that crawls the web is not the same thing as the Claude chatbot you talk to. Both come from Anthropic, but they do very different jobs.
What Is ClaudeBot and What Does It Do?
ClaudeBot is an automated program, often called a crawler or spider, that follows links across the public internet and reads the content of the pages it finds. Anthropic uses the data it collects as part of the material that can help train and improve its AI models.
In practice, ClaudeBot works much like other well known crawlers such as Googlebot or Bingbot. It requests a page, reads the visible text, and moves on. It focuses on publicly available pages and is meant to respect the rules you set in your robots.txt file.
Here is what the ClaudeBot crawler typically does:
- Visits public web pages by following links and sitemaps.
- Reads page text so it can be used as web data by Anthropic.
- Checks your robots.txt file to see which paths it is allowed to access.
- Identifies itself with a clear user agent string so you can recognize it in logs.
The ClaudeBot User Agent String
Every crawler identifies itself with a user agent, a short line of text sent with each request. The ClaudeBot user agent makes it easy to spot in your access logs and to target in your robots.txt rules.
The token you match against is simply ClaudeBot. A full user agent line from the crawler generally looks like this:
| Crawler | User agent token | Example full string |
|---|---|---|
| ClaudeBot | ClaudeBot | Mozilla/5.0 (compatible; ClaudeBot/1.0; +http://www.anthropic.com/claudebot) |
The exact version number and details can change over time, so the reliable approach is to match the word ClaudeBot rather than a full fixed string. Anthropic has also used other agent names for different purposes, such as fetching a single page a user asked about, so it is worth reviewing Anthropic's own crawler documentation if you want to handle each one separately.
One caution: a user agent can be copied by anyone, so a request claiming to be ClaudeBot is not proof on its own. If you need certainty, verify the request against the network ranges Anthropic publishes rather than trusting the name alone. This guide does not list specific IP addresses, because those can change and should always be taken from the official source.
How to Allow ClaudeBot in robots.txt
By default, if your robots.txt file does not block it, ClaudeBot is allowed to crawl your public pages. If you want to make that permission explicit, or keep certain areas open while closing others, you can name the crawler directly.
To clearly allow ClaudeBot across your whole site, add this to the robots.txt file at the root of your domain:
- User-agent: ClaudeBot
- Allow: /
Allowing the crawler means your content can be read by Anthropic as part of its web data. Some publishers welcome this for visibility, while others prefer to keep their material out. There is no single right answer; it depends on your goals for the site.
How to Block ClaudeBot in robots.txt
If you would rather keep your pages out, the simplest method is a robots.txt rule. This is the standard answer to the question of how to block ClaudeBot, and it takes just two lines.
Place the following at the root of your domain, for example at yoursite.com/robots.txt:
- User-agent: ClaudeBot
- Disallow: /
The Disallow: / line tells the crawler not to access any path on your site. You can also block only part of your site by listing specific folders, for example Disallow: /members/, while leaving the rest open.
A few important points to keep in mind:
- robots.txt is a request, not a lock. Well behaved crawlers follow it, but it does not physically stop access. For truly private material, use passwords or server level restrictions.
- Changes are not instant. A crawler only sees your new rule the next time it reads your robots.txt, so it can take time to take effect.
- Blocking ClaudeBot does not affect search engines. Googlebot and Bingbot use their own separate rules, so blocking the Anthropic crawler will not remove you from Google or Bing.
ClaudeBot vs the Claude Chatbot
This is where a lot of confusion starts. ClaudeBot is a background crawler that gathers web data. The Claude chatbot is the assistant people chat with to write, summarize, and answer questions. They share a parent company but are not the same product.
If you want to understand the assistant side, our guide on what is Claude AI walks through what the chatbot can do, and Claude models explained covers the different model versions. To learn about the company itself, see what is Anthropic.
The short version: seeing ClaudeBot in your logs means the crawler visited your site, not that a person used the chatbot on your pages. Blocking the crawler does not stop anyone from using the Claude app, and using the Claude app does not send this crawler to your site.
Should You Allow or Block the Anthropic Crawler?
There is no universal answer, but a few simple questions can guide your decision about the Anthropic crawler.
- Do you want broad reach? Allowing the crawler keeps your content in the pool of public web data.
- Is your content sensitive or paid? If so, blocking the crawler, and protecting the pages properly, makes sense.
- Are you worried about server load? Reputable crawlers are usually light, but you can block or limit any bot that requests too aggressively.
If you also publish content and want to check how machine generated text reads, our AI content detector guide is a useful companion. And if you are choosing which assistant to actually use day to day, the Claude AI chat overview compares the experience.
Try Claude and Other Models in Chatgbot
Understanding ClaudeBot is one thing; actually using Claude is another. If you want to use the Claude assistant without committing to a single provider, Chatgbot lets you use Claude alongside other leading models, including GPT, DeepSeek, Qwen, and GLM, in one place for one subscription. You can switch models mid conversation, so you are never locked into a single choice.
Frequently Asked Questions
Is ClaudeBot the same as the Claude chatbot?
No. ClaudeBot is Anthropic's web crawler that reads public pages to gather web data. The Claude chatbot is the assistant people talk to. They come from the same company but do different jobs.
How do I stop ClaudeBot from crawling my site?
Add a rule to your robots.txt file with User-agent: ClaudeBot on one line and Disallow: / on the next. This asks the crawler to stay off your pages. For private data, also protect the pages with passwords or server restrictions.
Will blocking ClaudeBot hurt my search rankings?
No. ClaudeBot is separate from search engine crawlers like Googlebot and Bingbot, which follow their own robots.txt rules. Blocking the Anthropic crawler does not remove your site from Google or Bing.
How can I confirm a request is really from ClaudeBot?
Check the ClaudeBot user agent string in your logs first, then verify the request against the network ranges Anthropic publishes. A user agent alone can be faked, so matching it to an official source is the reliable way to confirm.
The Bottom Line
ClaudeBot is a straightforward web crawler from Anthropic, easy to spot by its user agent and easy to allow or block through robots.txt. Decide based on your own goals for your content. And if you want to put Claude to work alongside other top models, Chatgbot is a simple place to start.