{"id":9812,"date":"2026-06-07T16:39:45","date_gmt":"2026-06-07T14:39:45","guid":{"rendered":"https:\/\/mybox.com\/help\/?post_type=manual_kb&#038;p=9812"},"modified":"2026-06-07T16:39:46","modified_gmt":"2026-06-07T14:39:46","slug":"how-to-use-the-robots-txt-file","status":"publish","type":"manual_kb","link":"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/","title":{"rendered":"How to Use the robots.txt File"},"content":{"rendered":"\n<div class=\"translation-block translation-block-merged\">\n<p class=\"wp-block-paragraph\">The <code>robots.txt<\/code> file allows you to provide instructions to search engine crawlers and other automated bots regarding which parts of your website should or should not be crawled.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It is one of the most commonly used files for basic search engine optimization (SEO) and website indexing management.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_86 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#What_Is_the_robotstxt_File\" >What Is the robots.txt File?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Basic_Syntax\" >Basic Syntax<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Allow_a_Bot_to_Crawl_a_Directory\" >Allow a Bot to Crawl a Directory<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Prevent_a_Bot_from_Crawling_a_Directory\" >Prevent a Bot from Crawling a Directory<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Specify_a_Sitemap\" >Specify a Sitemap<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Common_Directives\" >Common Directives<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#User-agent\" >User-agent<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Wildcard\" >Wildcard<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Disallow\" >Disallow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Allow\" >Allow<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Sitemap\" >Sitemap<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Where_Should_the_robotstxt_File_Be_Located\" >Where Should the robots.txt File Be Located?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Recommended_robotstxt_for_WordPress\" >Recommended robots.txt for WordPress<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Why_Include_a_Sitemap\" >Why Include a Sitemap?<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Posts\" >Posts<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Pages\" >Pages<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Testing_Your_robotstxt_File\" >Testing Your robots.txt File<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-18\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Important_Limitations\" >Important Limitations<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-19\" href=\"https:\/\/mybox.com\/help\/en\/knowledgebase\/how-to-use-the-robots-txt-file\/#Summary\" >Summary<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"What_Is_the_robotstxt_File\"><\/span>What Is the robots.txt File?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<\/div>\n\n<div id=\"mybox-2921991307\" class=\"mybox-content mybox-entity-placement\"><div class=\"early-access-banner-inpost\">\r\n  <div class=\"banner-left-inpost\">\r\n    <div class=\"icon-box-inpost\">\r\n      <img decoding=\"async\" src=\"https:\/\/mybox.com\/help\/wp-content\/uploads\/2026\/02\/square-info-icon.svg\" alt=\"Info\">\r\n    <\/div>\r\n    <div class=\"text-box-inpost\">\r\n      <span class=\"label-inpost\"><span class=\"translation-block translation-block-banner-text\">Early access<\/span><\/span>\r\n      <h4><span class=\"translation-block translation-block-banner-text\">Still need help?<\/span><\/h4>\r\n      <p><span class=\"translation-block translation-block-banner-text\">Contact our customer service team.<\/span><\/p>\r\n    <\/div>\r\n  <\/div>\r\n\r\n  <div class=\"banner-right-inpost\">\r\n    <a href=\"https:\/\/panel.mybox.com\/helpdesk2\/v\/list\/\" class=\"banner-button-inpost\"><span class=\"translation-block translation-block-banner-text\">Message us<\/span><\/a>\r\n  <\/div>\r\n<\/div><\/div>\n\n<div class=\"translation-block translation-block-merged\"><p class=\"wp-block-paragraph\">The <code>robots.txt<\/code> file contains rules that are intended exclusively for web crawlers.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Unlike <code>.htaccess<\/code>, the <code>robots.txt<\/code> file:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Does not affect how visitors access your website.<\/li>\n\n\n\n<li>Does not provide security protection.<\/li>\n\n\n\n<li>Does not block access to files or directories.<\/li>\n\n\n\n<li>Only provides guidance to crawlers that choose to respect its rules.<\/li>\n<\/ul>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\"><strong>Important<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The robots.txt file relies on crawler compliance. While major search engines such as Google and Bing respect its rules, malicious bots may ignore them entirely.<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Basic_Syntax\"><\/span>Basic Syntax<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Allow_a_Bot_to_Crawl_a_Directory\"><\/span>Allow a Bot to Crawl a Directory<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: BotName\nAllow: \/directory\/\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Prevent_a_Bot_from_Crawling_a_Directory\"><\/span>Prevent a Bot from Crawling a Directory<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: BotName\nDisallow: \/directory\/\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Specify_a_Sitemap\"><\/span>Specify a Sitemap<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>Sitemap: https:\/\/yourdomain.com\/sitemap.xml\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Common_Directives\"><\/span>Common Directives<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"User-agent\"><\/span>User-agent<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Defines which crawler the rule applies to.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: Googlebot\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Wildcard\"><\/span>Wildcard<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Applies the rule to all crawlers:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: *\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Disallow\"><\/span>Disallow<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Prevents crawling of a specific path:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Disallow: \/private\/\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Allow\"><\/span>Allow<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Explicitly allows crawling of a path:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Allow: \/images\/\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Sitemap\"><\/span>Sitemap<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Provides the location of your XML sitemap:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Sitemap: https:\/\/yourdomain.com\/sitemap.xml\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Where_Should_the_robotstxt_File_Be_Located\"><\/span>Where Should the robots.txt File Be Located?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The file must be placed in the root directory of your website.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">By default, the location is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>public_html\/robots.txt\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Example directory structure:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>public_html\n\u251c\u2500\u2500 robots.txt\n\u251c\u2500\u2500 index.php\n\u251c\u2500\u2500 wp-content\n\u2514\u2500\u2500 wp-admin\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Once uploaded, the file should be accessible at:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>https:&#47;&#47;yourdomain.com\/robots.txt\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Recommended_robotstxt_for_WordPress\"><\/span>Recommended robots.txt for WordPress<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A commonly used configuration for WordPress websites is:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-Agent: *\n\nAllow: \/wp-content\/uploads\/\n\nDisallow: \/wp-content\/plugins\/\nDisallow: \/wp-admin\/\nDisallow: \/readme.html\nDisallow: \/refer\/\n\nSitemap: https:\/\/yourdomain.com\/post-sitemap.xml\nSitemap: https:\/\/yourdomain.com\/page-sitemap.xml\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Replace:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>yourdomain.com\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">with your actual domain name.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Why_Include_a_Sitemap\"><\/span>Why Include a Sitemap?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A sitemap helps search engines discover and index your content more efficiently.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Typical WordPress sitemaps include:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Posts\"><\/span>Posts<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>https:&#47;&#47;yourdomain.com\/post-sitemap.xml\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Pages\"><\/span>Pages<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>https:&#47;&#47;yourdomain.com\/page-sitemap.xml\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Many SEO plugins generate these automatically.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Popular options include:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Yoast SEO<\/li>\n\n\n\n<li>Rank Math<\/li>\n\n\n\n<li>All in One SEO<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Testing_Your_robotstxt_File\"><\/span>Testing Your robots.txt File<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">After creating the file:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Upload it to the website root directory.<\/li>\n\n\n\n<li>Visit:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>https:&#47;&#47;yourdomain.com\/robots.txt\n<\/code><\/pre>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li>Verify that the file contents are displayed correctly.<\/li>\n<\/ol>\n\n\n\n<p class=\"wp-block-paragraph\">You can also test the file using search engine webmaster tools to confirm that the rules are being interpreted as expected.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Important_Limitations\"><\/span>Important Limitations<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The robots.txt file:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Does not secure sensitive content.<\/li>\n\n\n\n<li>Does not prevent direct access to files.<\/li>\n\n\n\n<li>Does not stop malicious crawlers.<\/li>\n\n\n\n<li>Should not be used as a security mechanism.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">For truly private content, use:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Password protection.<\/li>\n\n\n\n<li>Authentication.<\/li>\n\n\n\n<li>Access restrictions via <code>.htaccess<\/code>.<\/li>\n\n\n\n<li>Application-level permissions.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>robots.txt<\/code> file allows you to control how search engines and other crawlers interact with your website.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To use it:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Create a file named:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>robots.txt\n<\/code><\/pre>\n\n\n\n<ol start=\"2\" class=\"wp-block-list\">\n<li>Place it in:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>public_html\n<\/code><\/pre>\n\n\n\n<ol start=\"3\" class=\"wp-block-list\">\n<li>Add rules using:\n<ul class=\"wp-block-list\">\n<li><code>User-agent<\/code><\/li>\n\n\n\n<li><code>Allow<\/code><\/li>\n\n\n\n<li><code>Disallow<\/code><\/li>\n\n\n\n<li><code>Sitemap<\/code><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Verify it is accessible at:<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>https:&#47;&#47;yourdomain.com\/robots.txt\n<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">A properly configured robots.txt file can help improve indexing efficiency and give search engines clearer instructions about which parts of your website should be crawled.<\/p>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"template":"","format":"standard","manualknowledgebasecat":[125],"manual_kb_tag":[3729,4781,4790,4791,4792,4793,4794,4795,1365,3723],"class_list":["post-9812","manual_kb","type-manual_kb","status-publish","format-standard","hentry","manualknowledgebasecat-websites","manual_kb_tag-robots-file","manual_kb_tag-user-agent","manual_kb_tag-web-crawlers","manual_kb_tag-website-indexing","manual_kb_tag-crawl-directives","manual_kb_tag-disallow-directive","manual_kb_tag-allow-directive","manual_kb_tag-sitemap-xml","manual_kb_tag-search-engine-optimization","manual_kb_tag-robots-txt"],"_links":{"self":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/9812","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb"}],"about":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/types\/manual_kb"}],"author":[{"embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/users\/1"}],"version-history":[{"count":1,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/9812\/revisions"}],"predecessor-version":[{"id":9813,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb\/9812\/revisions\/9813"}],"wp:attachment":[{"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/media?parent=9812"}],"wp:term":[{"taxonomy":"manualknowledgebasecat","embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manualknowledgebasecat?post=9812"},{"taxonomy":"manual_kb_tag","embeddable":true,"href":"https:\/\/mybox.com\/help\/en\/wp-json\/wp\/v2\/manual_kb_tag?post=9812"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}