{"id":7711,"date":"2026-04-15T10:15:02","date_gmt":"2026-04-15T08:15:02","guid":{"rendered":"https:\/\/mybox.com\/help\/?post_type=manual_kb&#038;p=7711"},"modified":"2026-04-15T10:15:03","modified_gmt":"2026-04-15T08:15:03","slug":"cloudflare-ai-crawl-control-full-control-over-ai-bots","status":"publish","type":"manual_kb","link":"https:\/\/mybox.com\/help\/en\/knowledgebase\/cloudflare-ai-crawl-control-full-control-over-ai-bots\/","title":{"rendered":"Cloudflare AI Crawl Control \u2013 Full control over AI bots"},"content":{"rendered":"\n<div class=\"translation-block translation-block-merged\">\n<p class=\"wp-block-paragraph\">In <strong>April 2026<\/strong>, the internet has moved from &#8220;Open vs. Blocked&#8221; to a <strong>&#8220;Permission-Based&#8221; economy<\/strong>. Cloudflare AI Crawl Control (formerly AI Audit) is the primary dashboard for publishers to manage how AI agents\u2014like those from OpenAI, Meta, and Anthropic\u2014interact with their content.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On <strong>mybox<\/strong>, this tool is essential because AI bots now account for roughly <strong>20\u201330% of total web traffic<\/strong>, and many of these &#8220;read&#8221; your work without ever sending a human visitor or clicking an ad.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_86 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/cloudflare-ai-crawl-control-full-control-over-ai-bots\/#1_The_%E2%80%9CPay_Per_Crawl%E2%80%9D_Marketplace_2026_New_Feature\" >1. The &#8220;Pay Per Crawl&#8221; Marketplace (2026 New Feature)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/cloudflare-ai-crawl-control-full-control-over-ai-bots\/#2_Cryptographic_Bot_Verification_Web_Bot_Auth\" >2. Cryptographic Bot Verification (Web Bot Auth)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/cloudflare-ai-crawl-control-full-control-over-ai-bots\/#3_Strategic_Configuration_on_mybox\" >3. Strategic Configuration on mybox<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/cloudflare-ai-crawl-control-full-control-over-ai-bots\/#4_Step-by-Step_Dashboard_Setup\" >4. Step-by-Step Dashboard Setup<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/cloudflare-ai-crawl-control-full-control-over-ai-bots\/#5_Performance_SEO_Considerations\" >5. Performance &amp; SEO Considerations<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/cloudflare-ai-crawl-control-full-control-over-ai-bots\/#Summary\" >Summary<\/a><\/li><\/ul><\/nav><\/div>\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"1_The_%E2%80%9CPay_Per_Crawl%E2%80%9D_Marketplace_2026_New_Feature\"><\/span><strong>1. The &#8220;Pay Per Crawl&#8221; Marketplace (2026 New Feature)<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<\/div>\n\n<div id=\"mybox-2832822321\" class=\"mybox-content mybox-entity-placement\"><div class=\"early-access-banner-inpost\">\r\n  <div class=\"banner-left-inpost\">\r\n    <div class=\"icon-box-inpost\">\r\n      <img decoding=\"async\" src=\"https:\/\/mybox.com\/help\/wp-content\/uploads\/2026\/02\/square-info-icon.svg\" alt=\"Info\">\r\n    <\/div>\r\n    <div class=\"text-box-inpost\">\r\n      <span class=\"label-inpost\"><span class=\"translation-block translation-block-banner-text\">Early access<\/span><\/span>\r\n      <h4><span class=\"translation-block translation-block-banner-text\">Still need help?<\/span><\/h4>\r\n      <p><span class=\"translation-block translation-block-banner-text\">Contact our customer service team.<\/span><\/p>\r\n    <\/div>\r\n  <\/div>\r\n\r\n  <div class=\"banner-right-inpost\">\r\n    <a href=\"https:\/\/panel.mybox.com\/helpdesk2\/v\/list\/\" class=\"banner-button-inpost\"><span class=\"translation-block translation-block-banner-text\">Message us<\/span><\/a>\r\n  <\/div>\r\n<\/div><\/div>\n\n<div class=\"translation-block translation-block-merged\"><p class=\"wp-block-paragraph\">The biggest shift this year is the <strong>Pay Per Crawl<\/strong> model. Instead of a simple &#8220;Yes\/No&#8221; toggle, Cloudflare now acts as the <strong>Merchant of Record<\/strong> for your data.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>HTTP 402 (Payment Required):<\/strong> When an AI bot attempts to crawl a protected section of your <strong>mybox<\/strong> site, Cloudflare serves an HTTP 402 status code.<\/li>\n\n\n\n<li><strong>Pricing &amp; Negotiation:<\/strong> You can set a price (e.g., $0.01 per page) in your dashboard. Compliant AI bots will see the price in the response headers, process the micro-payment via Stripe\/Cloudflare, and then gain access to the content.<\/li>\n\n\n\n<li><strong>Why Use It?<\/strong> It turns &#8220;Data Theft&#8221; into a <strong>direct revenue stream<\/strong>. If a multi-billion dollar AI company wants to train on your specialized archives, they finally have a technical way to pay you for it.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"2_Cryptographic_Bot_Verification_Web_Bot_Auth\"><\/span><strong>2. Cryptographic Bot Verification (Web Bot Auth)<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">In 2026, simple &#8220;User-Agent&#8221; strings (like saying &#8220;I am GPTBot&#8221;) are no longer trusted because they are easily spoofed.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>The Standard:<\/strong> Cloudflare uses the <strong>Web Bot Auth<\/strong> (an IETF draft standard).<\/li>\n\n\n\n<li><strong>How it Works:<\/strong> Legitimate AI bots must cryptographically sign their requests. Cloudflare verifies these signatures against a global registry of &#8220;Verified Bots.&#8221;<\/li>\n\n\n\n<li><strong>On mybox:<\/strong> This ensures that when you &#8220;Allow&#8221; an AI Search bot (like Perplexity), it&#8217;s actually that bot and not a malicious scraper pretending to be one.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"3_Strategic_Configuration_on_mybox\"><\/span><strong>3. Strategic Configuration on mybox<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Don&#8217;t use a global &#8220;Block All&#8221; strategy. Use a <strong>Layered Defense<\/strong> based on content value:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><td><strong>Content Type<\/strong><\/td><td><strong>Recommendation<\/strong><\/td><td><strong>Implementation<\/strong><\/td><\/tr><\/thead><tbody><tr><td><strong>Public Blog\/News<\/strong><\/td><td><strong>Allow (with Attribution)<\/strong><\/td><td>Allow &#8220;AI Search&#8221; bots that provide citations and links back to your site.<\/td><\/tr><tr><td><strong>Premium \/ Archive<\/strong><\/td><td><strong>Pay Per Crawl (402)<\/strong><\/td><td>Force AI companies to pay to scrape your historical deep-dives or data.<\/td><\/tr><tr><td><strong>API \/ Dynamic Feeds<\/strong><\/td><td><strong>Block (403)<\/strong><\/td><td>Block AI bots to save your <strong>mybox<\/strong> server resources and API credits.<\/td><\/tr><tr><td><strong>User Data \/ Admin<\/strong><\/td><td><strong>Hard Block<\/strong><\/td><td>Ensure these paths are blocked globally to prevent &#8220;Data Leakage&#8221; into AI models.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"4_Step-by-Step_Dashboard_Setup\"><\/span><strong>4. Step-by-Step Dashboard Setup<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<ol start=\"1\" class=\"wp-block-list\">\n<li><strong>Navigate:<\/strong> Go to <strong>Security > AI Crawl Control<\/strong> (or <strong>AI Audit<\/strong>) in your Cloudflare dashboard.<\/li>\n\n\n\n<li><strong>Audit First:<\/strong> Look at the <strong>Crawlers Tab<\/strong>. You will see exactly which bots are visiting, their volume, and whether they respect your <code>robots.txt<\/code>.<\/li>\n\n\n\n<li><strong>One-Click Block:<\/strong> Use the global toggle for &#8220;Stealth&#8221; or &#8220;Non-Compliant&#8221; crawlers. These are bots that try to hide their identity.<\/li>\n\n\n\n<li><strong>Set Pricing:<\/strong> If you are on a Pro\/Business\/Enterprise plan, go to <strong>Settings > Pay Per Crawl<\/strong> to set your price per 1,000 requests.<\/li>\n\n\n\n<li><strong>Monitor Referrals:<\/strong> Check the <strong>Referrals Over Time<\/strong> chart. This shows if the AI bots you <em>do<\/em> allow are actually sending human traffic back to your site.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"5_Performance_SEO_Considerations\"><\/span><strong>5. Performance &amp; SEO Considerations<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Robots.txt Health:<\/strong> Cloudflare now provides a &#8220;Robots.txt Health&#8221; monitor. Ensure your <code>robots.txt<\/code> isn&#8217;t accidentally blocking AI bots that you <em>want<\/em> to index your site for citations (like Google\u2019s AI Overviews).<\/li>\n\n\n\n<li><strong>LCP Impact:<\/strong> Unlike traditional Bot Fight Mode, AI Crawl Control runs at the <strong>Edge<\/strong>. This means the block happens before the bot ever hits your <strong>mybox<\/strong> server, saving you bandwidth and improving the <strong>LCP<\/strong> for your real human visitors.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span><strong>Summary<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Cloudflare AI Crawl Control in 2026 puts you back in the driver&#8217;s seat. It moves beyond &#8220;Digital Fences&#8221; into a <strong>&#8220;Digital Marketplace&#8221;<\/strong> where you can finally monetize the AI bots that are reading your content. By leveraging <strong>HTTP 402<\/strong> and <strong>Web Bot Auth<\/strong>, you protect your <strong>mybox<\/strong> server while turning a privacy challenge into a business opportunity.<\/p>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"template":"","format":"standard","manualknowledgebasecat":[12],"manual_kb_tag":[210,242,587,771,850,936,964,3119,3405],"class_list":["post-7711","manual_kb","type-manual_kb","status-publish","format-standard","hentry","manualknowledgebasecat-cloudflare","manual_kb_tag-edge","manual_kb_tag-mybox","manual_kb_tag-blog","manual_kb_tag-server-resources","manual_kb_tag-http","manual_kb_tag-cloudflare","manual_kb_tag-dashboard","manual_kb_tag-configuration","manual_kb_tag-cloudflare-dashboard"],"_links":{"self":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/7711","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb"}],"about":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/types\/manual_kb"}],"author":[{"embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/users\/1"}],"version-history":[{"count":1,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/7711\/revisions"}],"predecessor-version":[{"id":7712,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/7711\/revisions\/7712"}],"wp:attachment":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/media?parent=7711"}],"wp:term":[{"taxonomy":"manualknowledgebasecat","embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manualknowledgebasecat?post=7711"},{"taxonomy":"manual_kb_tag","embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb_tag?post=7711"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}