AI crawlers can consume bandwidth, increase server load, and collect your content without sending visitors back to your website. Blocking them isn’t as simple as adding a few lines to robots.txt, since not every crawler respects those instructions.
You also need to distinguish between different types of AI bots.
- Training crawlers such as GPTBot, ClaudeBot, and Bytespider collect content for AI development.
- User-triggered agents, such as ChatGPT-User and Perplexity-User, may help your pages appear as cited sources in AI-generated answers.
Blocking every AI-related user agent could reduce your visibility as well as your server traffic.
This guide explains how to control AI crawler access without accidentally blocking legitimate search engines or useful AI referral traffic. You will learn how to:
- Identify the AI crawlers you may want to block
- Set rules in robots.txt
- block requests at the NGINX server level
- Configure Cloudflare’s AI crawler controls
- Test your rules and confirm they work
By the end, you will have a layered approach that gives you greater control over who can access your content and how much server capacity automated crawlers consume.
Why robots.txt Alone Won’t Stop AI Crawlers
AI-related crawlers serve several different purposes. Some collect content for model training, while others build search indexes or retrieve pages in response to a user request. High-volume crawling can consume bandwidth and server resources, but the impact depends on the crawler, its request rate, your caching configuration, and your application stack.
This is exactly why you should identify the crawler and its purpose before deciding whether to block it.
While adding a disallow directive to your robots.txt file is the traditional method for managing bots, it is ineffective against these bots because it runs on an honor system. Aggressive scrapers, poorly configured rogue bots, and many proprietary data brokers routinely ignore robots.txt entirely.
If you want to protect your infrastructure and intellectual property, you need to implement server-level blocking using NGINX or implement other firewall measures. This allows you to intercept these unauthorized requests at the server level, where you can drop the connection by returning a 403 Forbidden response before the request ever reaches your web application or consumes your server’s computing resources.
Suggested read: How To Use ModSecurity and OWASP CRS For Web App Firewall (WAF) To Secure Your Website
Which AI Bots Should You Block in 2026?
If you want to protect your server resources and intellectual property, you should prioritize blocking the following primary training crawlers.
Targeting High-Volume AI Training Crawlers
The following control tokens and user agents are associated primarily with AI training, model development, or large-scale dataset collection:
- GPTBot
- ClaudeBot
- Bytespider
- Amazonbot
- Applebot-Extended
- Google-Extended
- Meta-ExternalAgent
- CCBot
Claude-SearchBot should be considered separately, as Anthropic uses it to support search results rather than for general model training.
PerplexityBot is also separate from bulk training crawlers. Perplexity states that it uses PerplexityBot to build its search index and surface links in Perplexity results, not to train foundation models. Blocking it may prevent your pages from appearing in Perplexity search results.
Evaluating Real-Time AI Search Agents
Real-time fetchers operate under a different set of rules than bulk scrapers. Instead of indiscriminately harvesting data, these digital assistants visit your server only when a human query explicitly triggers them to fetch your content. This makes them an important component for maintaining visibility in modern, AI-driven search results.
Integrating with these tools can be a good marketing move, but it requires balancing visibility against your specific privacy concerns.
- ChatGPT-User
- Perplexity-User
- OAI-SearchBot
- Claude-User
Allowing these agents enables AI services to retrieve and cite your current content in response to user requests. It doesn’t guarantee that your page will be selected, cited, ranked prominently, or visited by the user. However, if your domain contains sensitive data or strictly paywalled content, you should weigh your goals carefully. If you prefer total privacy over AI search exposure, block these alongside the bulk training bots.
Anthropic uses separate user agents for different purposes. ClaudeBot crawls content for model development, Claude-SearchBot supports Claude’s search results, and Claude-User may retrieve a page in response to an individual user’s request.
Suggested read: How to Use Cloudflare Firewall Rules to Protect Your Web Application
Will Blocking AI Crawlers Hurt Your Google Rankings?
Blocking AI crawlers won’t harm your traditional Google Search rankings if you implement your security blocks correctly and target the right bots. Google designed its ecosystem to allow web admins to opt out of generative AI training without sacrificing their organic SEO visibility.
Google uses the Googlebot user agents to crawl, index, and rank your pages in standard search engine results. For AI training and generative model grounding (such as feeding live search context to Gemini), Google uses a separate control token known as Google-Extended. Disallowing Google-Extended tells Google not to use your content for training and grounding in certain Gemini systems. Google states that Google-Extended doesn’t affect inclusion or ranking in Google Search.
Google-Extended doesn’t control AI features that form part of Google Search itself. Access to those features is governed through Googlebot and standard Search controls such as nosnippet, data-nosnippet, max-snippet, and noindex.
However, there is a massive technical caveat for web admins: unlike most web scrapers, Google-Extended doesn’t have its own separate HTTP request user-agent string. The physical crawling is still executed using standard Google crawler user agents, meaning the Google-Extended token functions purely in a control capacity within your robots.txt file.
Because the phrase “Google-Extended” will never appear in your server’s incoming HTTP requests, you can’t block it at the NGINX or Cloudflare WAF level using a simple User-Agent string match. You must use the traditional robots.txt opt-out method for Google’s AI training and rely on server-level blocks for other companies’ bots.
Because of this shared infrastructure, you must be extremely cautious when writing your NGINX rules or WAF policies. Using broad wildcards like *Google* or *bot* to stop scrapers will result in a catastrophic SEO failure, as you will accidentally block the legitimate Googlebot, Googlebot-Image, and Googlebot-Video indexers that your business relies on.
Suggested read: Linux Server Hardening: 11 Steps to Secure a Production VPS
Implementing a Multi-Layered Strategy for AI Bot Management
Relying on a single method for blocking unwanted AI traffic can leave your infrastructure exposed. If you want to build a truly resilient defense, it is best to implement a layered strategy that covers public directives, server-level interception, and network-edge filtering. This ensures that if one mechanism fails or is ignored, your subsequent layers act as a fail-safe to protect your server resources.
Layer 1 – Block AI Bots with robots.txt
To set up your first layer of defense, create or edit a simple text file named robots.txt and place it in the root directory of your website (usually the public_html or htdocs folder).
To edit your robots.txt, you can simply open your web host’s file manager, create this file, and copy-paste the code below to ask all major 2026 AI training bots to stay away from your entire site:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Google-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /Why robots.txt alone is not enough (RFC 9309 is advisory)
While adding a robots.txt file is the industry standard first step, it is important to understand that it operates entirely on an honor system. RFC 9309 standardizes how crawlers should interpret robots.txt, but it doesn’t provide authentication or access control. Compliant crawlers follow its directives voluntarily, while another client can still request the same URL directly.
Reputable companies like Google and OpenAI currently program their bots to respect these rules, but aggressive scrapers, rogue data brokers, and malicious AI crawlers will completely ignore your robots.txt file and scrape your content anyway. Therefore, relying on this file alone leaves your server vulnerable to heavy automated traffic, so you must implement server-level blocking as well.
Layer 2 – Block AI Bots at the NGINX Layer
If you search for tutorials on how to block bots using NGINX, almost every old guide will tell you to place an if statement directly inside your server block (for example, if ($http_user_agent ~* "GPTBot") { return 403; }).
Complex rewrite logic inside an NGINX if block can produce unexpected results, particularly when it is placed inside a location block. A simple condition that performs only a return is much less problematic.
If you want to block traffic, you should use the map directive, which is a better and highly optimized alternative to if statements. When a visitor sends a request, NGINX checks their User-Agent exactly once against this map and assigns a variable (like a simple true/false flag). Because it is evaluated outside the complex location rules, it is incredibly fast and can automatically apply to every single website hosted on your entire server without repeating code.
To implement this on your server, you will need to access your global NGINX configuration (often located in /etc/nginx/conf.d/) and create a new file named ai_bot_map.conf. Paste the following map directive into that file, which flags the 2026 AI bots with a 1 (true) if they match, and a 0 (false) for normal human visitors:
map $http_user_agent $is_ai_bot {
default 0;
"~*GPTBot" 1;
"~*ClaudeBot" 1;
"~*Bytespider" 1;
"~*Amazonbot" 1;
"~*Meta-ExternalAgent" 1;
"~*CCBot" 1;
}Don’t add Google-Extended or Applebot-Extended to this map. They are robots.txt control tokens rather than independent crawler User-Agent strings. Instead, use robots.txt to control them.
Choosing the response code
After identifying the crawlers, you need to decide how your server should handle them when they arrive. Here is a list of appropriate HTTP status codes you can return when rejecting a web request.
403 Forbidden
Returning a 403 Forbidden status is the most common and standard choice because it clearly communicates to the bot that the server understood the request but is actively refusing to fulfill it.
444 (Nginx-specific, drops the connection with no response)
If you are dealing with an incredibly aggressive bot that is hitting your server thousands of times a minute and draining your bandwidth, you should use NGINX’s special 444 response code. Unlike standard HTTP codes, 444 doesn’t send any headers or error pages to the bot; it simply closes the connection immediately. This can reduce the response work and outbound data associated with rejected requests, although NGINX must still accept and process the connection far enough to match the rule.
410 Gone (signals permanent removal, deters re-crawls)
Another good option is to return a 410 Gone status code, which signals to the AI crawler that the resource it is looking for has been permanently deleted and won’t be returned. Use 410 Gone only when the requested resource has been permanently removed. Don’t return 410 solely as a bot-blocking technique for pages that remain available to other visitors. Use 403 or 444 when you are refusing access based on the requester rather than the state of the resource.
Logging Blocked Requests for Audit
You shouldn’t blindly block traffic without keeping a record of what your server is doing. You can instruct NGINX to log blocked bot attempts to a separate file so they don’t clutter your main website analytics.
By adding a simple directive like access_log /var/log/nginx/blocked_ai_bots.log; inside the block that returns your 403 or 444 code, you create an isolated, easily readable audit trail where you can safely monitor which AI companies are trying to harvest your data.
Suggested read: How to Block IP Address Using WordPress .htaccess File to Stop Bad Visitors
Layer 3 – Block AI Bots at the Edge with Cloudflare WAF
If you manage your DNS through Cloudflare, you can easily intercept AI bots at the network edge before they ever reach your web server by configuring their built-in bot policies. Cloudflare has updated its approach to AI traffic by introducing new features that categorize bots into three distinct behaviors:
- “Search” (crawlers indexing content to answer queries later and drive referrals)
- “Agent” (real-time automated bots acting on a human’s behalf, like chat fetch bots)
- “Training” (scrapers permanently absorbing your data to train large language models).
Configure AI Bot Policies
Cloudflare provides two related ways to manage AI traffic:
In AI Crawl Control, open the Crawlers tab to review individual crawlers and set each one to Allow or Block.
For broader behavior-based controls, open the Security settings and configure how Cloudflare handles Search, Agent, and Training traffic. Each category can be allowed, blocked across the domain, or blocked only on pages where Cloudflare detects advertising.
For each type of bot, you can choose to “Block (on all pages),” “Allow (do not block),” or select “Block on pages with ads,” which uses Cloudflare’s automated detection to block bots strictly on monetized pages while leaving the rest of your site accessible.
Additionally, from September 15, 2026, Cloudflare will automatically block ‘Training’ and ‘Agent’ bots on pages with ads for all new domains, while keeping Search bots allowed. Cloudflare’s updated system will apply the most restrictive rule to multi-purpose crawlers, such as Googlebot or Applebot, that crawl for both search indexing and AI training.
From 15 September 2026, Cloudflare plans to evaluate multi-purpose crawlers against all their declared behaviors. A crawler that combines Search and Training may therefore be affected by your Training rule even when Search traffic is allowed.
Review Cloudflare’s displayed outcome before applying category-wide rules, particularly where a crawler also supports conventional search discovery.
Before enforcing broad blocks across your entire domain, you can also establish governance rules using Cloudflare’s AI Audit tools. The AI Audit dashboard provides visibility into which AI services are scanning your website, allowing you to clearly see the volume of requests from specific bots and understand how they interact with your content.
By analyzing this traffic, novice users can make data-driven security decisions, such as explicitly allowing search-focused tools that drive referral traffic, while firmly rejecting exploitative scrapers.
Future of AI Content Monetization
Cloudflare is also testing Pay Per Crawl, which is currently in closed beta. Participating site owners can set a price and choose whether to allow, block, or charge supported crawlers.
A crawler that doesn’t provide the required payment information receives an HTTP 402 Payment Required response. Availability and crawler participation remain limited, so this shouldn’t yet be treated as a general replacement for blocking.
Suggested read: What is Fail2Ban with Setup & Configuration? (Detailed Guide)
Verify that NGINX Blocking is Working
After deploying your NGINX mapping rules, you shouldn’t simply assume they are functioning perfectly. Server-level configurations are powerful but unforgiving; a minor syntax error could potentially block legitimate traffic or fail to stop the unwanted bots you are targeting.
You need to validate your setup to ensure your defenses are correct and not causing collateral damage to your site’s availability. Follow the steps below to verify that your block list is successfully intercepting requests.
Curl with a faked GPTBot User-Agent and expect 403
The easiest way for a novice to test if their server block is working is to pretend to be an AI bot using the command line. Open your computer’s terminal (or Command Prompt) and type the following command exactly:
curl -I -A "GPTBot/1.0" https://example.comThis command sends a fake request to your site claiming to be GPTBot; if your NGINX rules are working correctly, your terminal will print out an HTTP/2 403 Forbidden error (or return an empty reply if you used the 444 code), proving the block is active.
Tail NGINX access logs and grep for the bot user-agents
To watch your server actively defend itself in real-time, you can filter your live server logs for specific bot names. Log in to your server via SSH and run the command:
tail -f /var/log/nginx/access.log | grep -i "gptbot"The tail -f command streams the log file live, while grep filters out everything except requests containing the word “gptbot”, allowing you to sit back and watch the exact moment the AI crawler hits your server and gets rejected.
Cross-check with Cloudflare bot analytics
Finally, you can verify your edge-level blocks by reviewing your visual data within the Cloudflare dashboard. Navigate to Security and then click on Events to see a complete log of all web traffic that triggered your WAF rules.
By filtering this list by “User Agent” or specifically looking at the “Block AI bots” rule metrics, you can visually confirm how many thousands of requests from scrapers like ClaudeBot and Bytespider were successfully dropped by Cloudflare’s network before they ever touched your origin server.
Final Thoughts: Take Control of Your Server Traffic
In this article, we have discussed how to manage custom NGINX configurations and protect your websites. It might sound overwhelming, but RunCloud makes the entire process effortless.
With RunCloud, you can easily manage and deploy sites on your own cloud infrastructure without needing to be a command-line expert. Its highly intuitive dashboard allows you to make quick NGINX updates and instantly reload your server to deploy those security rules across all your hosted applications in just a few clicks.
Whether you’re hosting a single high-traffic web application or managing dozens of client sites, having complete, frictionless control over your hosting environment is important.
Should You Block AI Crawlers? Common Questions Answered
Will blocking GPTBot or ClaudeBot affect my Google search rankings?
No, blocking AI crawlers like GPTBot or ClaudeBot won’t negatively impact your Google search rankings. These bots are completely separate from Googlebot, which is the scraper responsible for indexing your site for search engine results. Blocking AI scrapers only prevents your content from being scraped to train their large language models.
What is the difference between ClaudeBot, Claude-SearchBot, and Claude-User?
ClaudeBot crawls web content for Anthropic’s model development. Claude-SearchBot supports Claude’s search results, while Claude-User may retrieve a page in response to an individual user’s request.
You may choose to block ClaudeBot while allowing Claude-SearchBot and Claude-User if you want to limit training access without removing your content from Claude’s search and user-directed retrieval features.
Can AI crawlers bypass robots.txt and NGINX blocks?
Reputable AI crawlers will respect robots.txt directives, but rogue scrapers or malicious bots can easily ignore them. NGINX blocks are significantly more powerful because they intercept and drop the connection at the server level based on the User-Agent or IP address before the site even loads. Custom NGINX rules provide stronger enforcement than robots.txt because they can reject matching requests before they reach the application. User-Agent matching is not foolproof, since another client can spoof a crawler’s name.
For stronger protection, combine NGINX rules with rate limiting, request logs, verified crawler IP ranges where available, and edge-level bot controls such as Cloudflare.
Should I block ChatGPT-User and Perplexity-User the same way as I do for GPTBot?
It depends on your goals, as GPTBot crawls your site globally for AI training, whereas ChatGPT-User and Perplexity-User act as real-time search agents triggered by active user prompts. Blocking these “User” bots will prevent those AI tools from summarizing or linking to your live pages in their chat interfaces. If you want your site to be cited as a live source in AI-generated answers, you should leave user bots alone and block only training bots.
Does blocking Google-Extended affect Googlebot or Google Ads?
No, blocking Google-Extended only prevents your site’s content from being used to train Google’s generative AI models, such as Gemini. It operates entirely independently from traditional search indexing and advertising systems. Disallowing Google-Extended doesn’t affect your inclusion or ranking in Google Search. It is separate from the Googlebot controls used for conventional Search crawling.
Avoid blocking broader Google user agents or IP ranges at the server or firewall level, since an overly broad rule could interfere with Google services that rely on those crawlers.
