{"id":13653,"date":"2026-10-01T11:16:39","date_gmt":"2026-10-01T09:16:39","guid":{"rendered":"https:\/\/mybox.com\/help\/?post_type=manual_kb&#038;p=13653"},"modified":"2026-10-01T11:16:48","modified_gmt":"2026-10-01T09:16:48","slug":"differences-between-ai-txt-llms-txt-and-tdmrep-in-practice","status":"publish","type":"manual_kb","link":"https:\/\/mybox.com\/help\/en\/knowledgebase\/differences-between-ai-txt-llms-txt-and-tdmrep-in-practice\/","title":{"rendered":"Differences Between ai.txt, llms.txt, and TDMRep in Practice"},"content":{"rendered":"\n<div class=\"translation-block translation-block-merged\">\n<p class=\"wp-block-paragraph\">ai.txt, llms.txt, and TDMRep are often mentioned in the same breath as robots.txt, but in practice they serve completely different purposes. TDMRep is a formal mechanism for objecting to text and data mining, in accordance with the European DSM Directive. ai.txt primarily serves as an informal policy governing the use of content by AI systems, while llms.txt is a map of the most important content you want to present to language models.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_86 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/differences-between-ai-txt-llms-txt-and-tdmrep-in-practice\/#TDMRep%E2%80%94A_Legal_Signal_for_Text_and_Data_Mining\" >TDMRep\u2014A Legal Signal for Text and Data Mining<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/differences-between-ai-txt-llms-txt-and-tdmrep-in-practice\/#aitxt_%E2%80%93_AI_content_usage_policy\" >ai.txt &#8211; AI content usage policy<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/differences-between-ai-txt-llms-txt-and-tdmrep-in-practice\/#llmstxt_%E2%80%93_a_content_map_for_language_models\" >llms.txt \u2013 a content map for language models<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/differences-between-ai-txt-llms-txt-and-tdmrep-in-practice\/#What_do_these_mechanisms_control_in_practice\" >What do these mechanisms control in practice?<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"TDMRep%E2%80%94A_Legal_Signal_for_Text_and_Data_Mining\"><\/span>TDMRep\u2014A Legal Signal for Text and Data Mining<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">TDMRep was defined by the W3C as a simple protocol allowing content owners to restrict or permit text and data mining in a machine-readable format. It uses the `tdm-reservation` and `tdm-policy` fields, which can be specified in HTML meta tags, HTTP headers, or via the `tdmrep.json` file in the `.well-known` directory. A tdm-reservation value set to 1 indicates that TDM rights are reserved, while 0 indicates no restrictions.<\/p>\n\n\n\n<\/div>\n\n<div id=\"mybox-2191893146\" class=\"mybox-content mybox-entity-placement\"><div class=\"early-access-banner-inpost\">\r\n  <div class=\"banner-left-inpost\">\r\n    <div class=\"icon-box-inpost\">\r\n      <img decoding=\"async\" src=\"https:\/\/mybox.com\/help\/wp-content\/uploads\/2026\/02\/square-info-icon.svg\" alt=\"Info\">\r\n    <\/div>\r\n    <div class=\"text-box-inpost\">\r\n      <span class=\"label-inpost\"><span class=\"translation-block translation-block-banner-text\">Early access<\/span><\/span>\r\n      <h4><span class=\"translation-block translation-block-banner-text\">Still need help?<\/span><\/h4>\r\n      <p><span class=\"translation-block translation-block-banner-text\">Contact our customer service team.<\/span><\/p>\r\n    <\/div>\r\n  <\/div>\r\n\r\n  <div class=\"banner-right-inpost\">\r\n    <a href=\"https:\/\/panel.mybox.com\/helpdesk2\/v\/list\/\" class=\"banner-button-inpost\"><span class=\"translation-block translation-block-banner-text\">Message us<\/span><\/a>\r\n  <\/div>\r\n<\/div><\/div>\n\n<div class=\"translation-block translation-block-merged\"><p class=\"wp-block-paragraph\">Unlike robots.txt, TDMRep does not control bots\u2019 physical access to resources. It specifies the rights to use content in text and data mining processes, including for training AI models. It is primarily a legal and licensing signal that can serve as an important argument in discussions with AI solution providers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"aitxt_%E2%80%93_AI_content_usage_policy\"><\/span>ai.txt &#8211; AI content usage policy<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">ai.txt is an evolving, informal standard for a file typically located at \/ai.txt. Its purpose is to describe the rules for AI systems\u2019 use of a website\u2019s content. In such a file, you can specify whether you accept the use of content for training models, summarizing, quoting, or generating responses, as well as any restrictions and expectations regarding attribution.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In practice, ai.txt serves as a \u201cconsent, not access\u201d layer\u2014it signals consent or lack thereof, rather than acting as a technical block. It does not replace the terms of service or server-level security measures, but it helps provide clear guidelines to entities that wish to use your content in a manner consistent with the owner\u2019s wishes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"llmstxt_%E2%80%93_a_content_map_for_language_models\"><\/span>llms.txt \u2013 a content map for language models<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">llms.txt is a text or Markdown file typically located at \/llms.txt, whose purpose is not to block bots but to curate content. Such a file usually contains the website\u2019s name, a short description, and a list of the most important subpages\u2014such as the main offering, documentation, FAQs, and key articles\u2014that are particularly useful for language models.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can think of it as a map or guide to the content you want the AI to understand well and, if applicable, cite. llms.txt does not replace sitemap.xml or robots.txt\u2014rather, it serves as a supplement that points models to approved, high-quality sources within your domain.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_do_these_mechanisms_control_in_practice\"><\/span>What do these mechanisms control in practice?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">TDMRep governs text and data mining rights\u2014it specifies whether data may be used in TDM processes, including AI model training, but does not physically block access to the site. ai.txt outlines the rules for AI systems\u2019 use of content, with an emphasis on consent for training, summarization, and other forms of processing. llms.txt, on the other hand, controls neither rights nor access\u2014it helps models find and understand the most important content on the site.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">All three mechanisms serve more as signals to \u201cwell-behaved\u201d systems than as hard security measures. Real control over traffic still lies in robots.txt, server configuration, firewalls, and any WAF solutions or CDN-side filters. Therefore, it\u2019s best to treat TDMRep, ai.txt, and llms.txt as supplements rather than replacements for traditional security and SEO tools.<\/p>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"template":"","format":"standard","manualknowledgebasecat":[10698],"manual_kb_tag":[11054,11055,11056,11057,11058,11059,11060,11061,11062,10988,11012,11051,11052,11053],"class_list":["post-13653","manual_kb","type-manual_kb","status-publish","format-standard","hentry","manualknowledgebasecat-other","manual_kb_tag-ai-policy-on-the-website","manual_kb_tag-tdmrep","manual_kb_tag-llms-txt-table-of-contents","manual_kb_tag-text-and-data-mining","manual_kb_tag-tdmrep-dsm","manual_kb_tag-ai-crawler-monitoring","manual_kb_tag-protecting-content-from-ai","manual_kb_tag-opt-out-from-ai-training","manual_kb_tag-ai-usage-policy","manual_kb_tag-llms-txt","manual_kb_tag-ai-visibility","manual_kb_tag-ai-txt","manual_kb_tag-differences-ai-txt-llms-txt-tdmrep","manual_kb_tag-robots-txt-vs-ai-txt"],"_links":{"self":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/13653","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb"}],"about":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/types\/manual_kb"}],"author":[{"embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/users\/1"}],"version-history":[{"count":2,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/13653\/revisions"}],"predecessor-version":[{"id":13759,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/13653\/revisions\/13759"}],"wp:attachment":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/media?parent=13653"}],"wp:term":[{"taxonomy":"manualknowledgebasecat","embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manualknowledgebasecat?post=13653"},{"taxonomy":"manual_kb_tag","embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb_tag?post=13653"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}