{"id":9814,"date":"2026-06-07T16:41:02","date_gmt":"2026-06-07T14:41:02","guid":{"rendered":"https:\/\/mybox.com\/help\/?post_type=manual_kb&#038;p=9814"},"modified":"2026-06-07T16:41:03","modified_gmt":"2026-06-07T14:41:03","slug":"denumiri-comune-ale-agentilor-de-utilizator-folosite-de-motoarele-de-cautare-si-de-site-urile-web-populare","status":"publish","type":"manual_kb","link":"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/","title":{"rendered":"Nume comune ale agen\u021bilor de utilizator utilizate de motoarele de c\u0103utare \u0219i de site-urile web populare"},"content":{"rendered":"<div class=\"translation-block translation-block-merged\">\n<p class=\"wp-block-paragraph\">When reviewing website logs, firewall activity, or traffic statistics, you may notice requests originating from automated crawlers, also known as bots or spiders.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">These bots are commonly used by search engines and online platforms to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Index website content.<\/li>\n\n\n\n<li>Generate search engine results.<\/li>\n\n\n\n<li>Create social media link previews.<\/li>\n\n\n\n<li>Analyze website structure.<\/li>\n\n\n\n<li>Collect publicly available information.<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n<div id=\"mybox-3642048962\" class=\"mybox-content mybox-entity-placement\"><div class=\"early-access-banner-inpost\">\r\n  <div class=\"banner-left-inpost\">\r\n    <div class=\"icon-box-inpost\">\r\n      <img decoding=\"async\" src=\"https:\/\/mybox.com\/help\/wp-content\/uploads\/2026\/02\/square-info-icon.svg\" alt=\"Info\">\r\n    <\/div>\r\n    <div class=\"text-box-inpost\">\r\n      <span class=\"label-inpost\"><span class=\"translation-block translation-block-banner-text\">Acces timpuriu<\/span><\/span>\r\n      <h4><span class=\"translation-block translation-block-banner-text\">Mai ave\u021bi nevoie de ajutor?<\/span><\/h4>\r\n      <p><span class=\"translation-block translation-block-banner-text\">Contacta\u021bi echipa noastr\u0103 de servicii pentru clien\u021bi.<\/span><\/p>\r\n    <\/div>\r\n  <\/div>\r\n\r\n  <div class=\"banner-right-inpost\">\r\n    <a href=\"https:\/\/panel.mybox.com\/helpdesk2\/v\/list\/\" class=\"banner-button-inpost\"><span class=\"translation-block translation-block-banner-text\">Trimite mesaj<\/span><\/a>\r\n  <\/div>\r\n<\/div><\/div>\n\n<div class=\"translation-block translation-block-merged\"><p class=\"wp-block-paragraph\">Each crawler identifies itself using a <strong>User-Agent<\/strong> string, which can help you determine whether unusual traffic originates from a legitimate service or a potentially malicious bot.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_86 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-custom ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Why_Is_This_Information_Useful\" >Why Is This Information Useful?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#How_to_Check_User-Agent_Information\" >How to Check User-Agent Information<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Popular_Search_Engine_Crawlers\" >Popular Search Engine Crawlers<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Google_%E2%80%93_Googlebot\" >Google \u2013 Googlebot<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Microsoft_Bing_%E2%80%93_Bingbot\" >Microsoft Bing \u2013 Bingbot<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Yahoo_%E2%80%93_Slurp\" >Yahoo \u2013 Slurp<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#DuckDuckGo_%E2%80%93_DuckDuckBot\" >DuckDuckGo \u2013 DuckDuckBot<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Baidu_%E2%80%93_Baiduspider\" >Baidu \u2013 Baiduspider<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Yandex_%E2%80%93_YandexBot\" >Yandex \u2013 YandexBot<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Other_Popular_Crawlers\" >Other Popular Crawlers<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Sogou_Spider\" >Sogou Spider<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#ExaLead_%E2%80%93_Exabot\" >ExaLead \u2013 Exabot<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Social_Media_Crawlers\" >Social Media Crawlers<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Facebook_Link_Preview_Bot\" >Facebook Link Preview Bot<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Analytics_and_Ranking_Crawlers\" >Analytics and Ranking Crawlers<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Alexa_Crawler\" >Alexa Crawler<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Verifying_a_Suspected_Googlebot\" >Verifying a Suspected Googlebot<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-18\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Blocking_or_Allowing_Crawlers\" >Blocking or Allowing Crawlers<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-19\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#robotstxt\" >robots.txt<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-20\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#htaccess\" >.htaccess<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-21\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Important_Security_Note\" >Important Security Note<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-22\" href=\"https:\/\/mybox.com\/help\/ro\/knowledgebase\/common-user-agent-names-used-by-search-engines-and-popular-websites\/#Summary\" >Summary<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Why_Is_This_Information_Useful\"><\/span>Why Is This Information Useful?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Knowing common User-Agent names can help you:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Analyze traffic spikes.<\/li>\n\n\n\n<li>Identify search engine crawlers.<\/li>\n\n\n\n<li>Verify indexing activity.<\/li>\n\n\n\n<li>Create firewall or security rules.<\/li>\n\n\n\n<li>Troubleshoot website performance.<\/li>\n\n\n\n<li>Detect suspicious activity masquerading as a search engine bot.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"How_to_Check_User-Agent_Information\"><\/span>How to Check User-Agent Information<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">User-Agent information can typically be found in:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Web server logs<\/li>\n\n\n\n<li>Access logs<\/li>\n\n\n\n<li>Analytics platforms<\/li>\n\n\n\n<li>Security tools<\/li>\n\n\n\n<li>Firewall reports<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">When investigating crawler activity, look for the User-Agent field in the request details.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Popular_Search_Engine_Crawlers\"><\/span>Popular Search Engine Crawlers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Google_%E2%80%93_Googlebot\"><\/span>Google \u2013 Googlebot<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Googlebot\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agent<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Mozilla\/5.0 (compatible; Googlebot\/2.1; +http:\/\/www.google.com\/bot.html)\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Microsoft_Bing_%E2%80%93_Bingbot\"><\/span>Microsoft Bing \u2013 Bingbot<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Bingbot\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agent<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Mozilla\/5.0 (compatible; Bingbot\/2.0; +http:\/\/www.bing.com\/bingbot.htm)\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Yahoo_%E2%80%93_Slurp\"><\/span>Yahoo \u2013 Slurp<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Slurp\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agent<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Mozilla\/5.0 (compatible; Yahoo! Slurp; http:\/\/help.yahoo.com\/help\/us\/ysearch\/slurp)\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"DuckDuckGo_%E2%80%93_DuckDuckBot\"><\/span>DuckDuckGo \u2013 DuckDuckBot<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>DuckDuckBot\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agent<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>DuckDuckBot\/1.0; (+http:\/\/duckduckgo.com\/duckduckbot.html)\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Baidu_%E2%80%93_Baiduspider\"><\/span>Baidu \u2013 Baiduspider<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Baiduspider\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agent<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Mozilla\/5.0 (compatible; Baiduspider\/2.0; +http:\/\/www.baidu.com\/search\/spider.html)\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Yandex_%E2%80%93_YandexBot\"><\/span>Yandex \u2013 YandexBot<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>YandexBot\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agent<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Mozilla\/5.0 (compatible; YandexBot\/3.0; +http:\/\/yandex.com\/bots)\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Other_Popular_Crawlers\"><\/span>Other Popular Crawlers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Sogou_Spider\"><\/span>Sogou Spider<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Common User-Agent examples:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Sogou Pic Spider\/3.0\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Sogou head spider\/3.0\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Sogou web spider\/4.0\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Sogou Orion spider\/3.0\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Sogou-Test-Spider\/4.0\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"ExaLead_%E2%80%93_Exabot\"><\/span>ExaLead \u2013 Exabot<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Exabot\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agents<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>Mozilla\/5.0 (compatible; Exabot\/3.0; +http:\/\/www.exabot.com\/go\/robot)\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>Mozilla\/5.0 (compatible; Konqueror\/3.5; Linux) KHTML\/3.5.5 (like Gecko) (Exabot-Thumbnails)\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Social_Media_Crawlers\"><\/span>Social Media Crawlers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Facebook_Link_Preview_Bot\"><\/span>Facebook Link Preview Bot<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When a link is shared on Facebook, Meta uses a crawler to retrieve page information and generate a preview.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>facebookexternalhit\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agents<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>facebookexternalhit\/1.0\n<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>facebookexternalhit\/1.1\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Analytics_and_Ranking_Crawlers\"><\/span>Analytics and Ranking Crawlers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Alexa_Crawler\"><\/span>Alexa Crawler<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Short Name<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>ia_archiver\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Example User-Agent<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>ia_archiver (+http:\/\/www.alexa.com\/site\/help\/webmasters; crawler@alexa.com)\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Verifying_a_Suspected_Googlebot\"><\/span>Verifying a Suspected Googlebot<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Not every bot that claims to be Googlebot is legitimate.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Some malicious crawlers intentionally spoof Googlebot User-Agent strings.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you need to verify whether a request actually originated from Google, Google provides documentation on validating crawler IP addresses.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can also use:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/developers.google.com\/search\/docs\/crawling-indexing\/verifying-googlebot?utm_source=chatgpt.com\" target=\"_blank\" rel=\"noopener\">Google Search Central \u2013 Verify Googlebot<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Blocking_or_Allowing_Crawlers\"><\/span>Blocking or Allowing Crawlers<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If necessary, crawler access can be managed through:<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"robotstxt\"><\/span>robots.txt<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: Googlebot\nDisallow:\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"htaccess\"><\/span>.htaccess<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>RewriteEngine On\nRewriteCond %{HTTP_USER_AGENT} BadBot &#91;NC]\nRewriteRule .* - &#91;F,L]\n<\/code><\/pre>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\">Note: Blocking legitimate search engine crawlers may negatively affect your website&#8217;s visibility in search results.<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Important_Security_Note\"><\/span>Important Security Note<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Not all bots visiting your website are beneficial.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">While search engine crawlers help index your content, some automated bots may:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Search for vulnerabilities.<\/li>\n\n\n\n<li>Attempt brute-force attacks.<\/li>\n\n\n\n<li>Scrape website content.<\/li>\n\n\n\n<li>Probe outdated software.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">To reduce risk:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Keep your CMS updated.<\/li>\n\n\n\n<li>Update plugins and themes regularly.<\/li>\n\n\n\n<li>Monitor access logs.<\/li>\n\n\n\n<li>Use security plugins and firewalls.<\/li>\n\n\n\n<li>Block known malicious bots when appropriate.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Summary\"><\/span>Summary<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">User-Agent strings allow you to identify the source of automated traffic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Some of the most common crawler names include:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Service<\/th><th>User-Agent<\/th><\/tr><\/thead><tbody><tr><td>Google<\/td><td>Googlebot<\/td><\/tr><tr><td>Bing<\/td><td>Bingbot<\/td><\/tr><tr><td>Yahoo<\/td><td>Slurp<\/td><\/tr><tr><td>DuckDuckGo<\/td><td>DuckDuckBot<\/td><\/tr><tr><td>Baidu<\/td><td>Baiduspider<\/td><\/tr><tr><td>Yandex<\/td><td>YandexBot<\/td><\/tr><tr><td>Facebook<\/td><td>facebookexternalhit<\/td><\/tr><tr><td>Alexa<\/td><td>ia_archiver<\/td><\/tr><tr><td>ExaLead<\/td><td>Exabot<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Understanding these identifiers can help you analyze traffic patterns, troubleshoot indexing issues, and distinguish legitimate crawlers from potentially malicious bots.<\/p>\n<\/div>","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"template":"","format":"standard","manualknowledgebasecat":[125],"manual_kb_tag":[3730,4781,4782,4783,4784,4785,4786,4787,4788,4789],"class_list":["post-9814","manual_kb","type-manual_kb","status-publish","format-standard","hentry","manualknowledgebasecat-websites","manual_kb_tag-search-engine-bots","manual_kb_tag-user-agent","manual_kb_tag-user-agent-string","manual_kb_tag-web-crawler","manual_kb_tag-search-engine-crawler","manual_kb_tag-googlebot","manual_kb_tag-bingbot","manual_kb_tag-slurp","manual_kb_tag-duckduckbot","manual_kb_tag-baiduspider"],"_links":{"self":[{"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/manual_kb\/9814","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/manual_kb"}],"about":[{"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/types\/manual_kb"}],"author":[{"embeddable":true,"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/users\/1"}],"version-history":[{"count":1,"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/manual_kb\/9814\/revisions"}],"predecessor-version":[{"id":9815,"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/manual_kb\/9814\/revisions\/9815"}],"wp:attachment":[{"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/media?parent=9814"}],"wp:term":[{"taxonomy":"manualknowledgebasecat","embeddable":true,"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/manualknowledgebasecat?post=9814"},{"taxonomy":"manual_kb_tag","embeddable":true,"href":"https:\/\/mybox.com\/help\/ro\/wp-json\/wp\/v2\/manual_kb_tag?post=9814"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}