The .htaccess file is an Apache configuration file that allows you to modify server behavior on a per-directory basis. One common use case is blocking unwanted bots and crawlers that generate unnecessary traffic, consume resources, or attempt malicious activity.
Before making changes to .htaccess, always create a backup of the existing file.
Table of Contents
Blocking a specific IP address
If a bot consistently connects from a single IP address, you can block it directly.
Deny from 111.222.333.444
Replace 111.222.333.444 with the IP address you want to block.
Blocking an entire IP range
If unwanted traffic originates from a specific network, you can block the entire range.
Deny from 111.222.333.0/24
This example blocks all addresses within the specified subnet.
Blocking bots by User-Agent
Many bots identify themselves through the User-Agent header. You can block a specific bot name using:
SetEnvIfNoCase User-Agent "badbot" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot
In this example, any request containing badbot in its User-Agent string will be denied.
Blocking multiple unwanted User-Agents
You can block several known bots in a single rule:
SetEnvIfNoCase User-Agent "(badbot|evilcrawler|spamcrawler)" bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot
Blocking suspicious User-Agent headers
Some automated tools use identifiable User-Agent strings.
Example:
SetEnvIfNoCase User-Agent ".*(libwww-perl|aesop_com_spiderman)" HTTP_SAFE_BADBOT
Deny from env=HTTP_SAFE_BADBOT
Requests matching these patterns will be blocked automatically.
Important considerations
Before implementing bot-blocking rules:
- verify that the bot is genuinely unwanted
- avoid blocking legitimate search engine crawlers
- test changes after editing
.htaccess - monitor server logs for unintended side effects
Incorrect rules may block legitimate visitors or affect website functionality.
Alternative approaches
Depending on your hosting environment, bot management can also be performed using:
- web application firewalls (WAF)
- CDN security services
- server-level firewall rules
- security plugins for CMS platforms
These solutions often provide more flexibility and easier management than .htaccess rules alone.
Summary
The .htaccess file can be used to block unwanted bots based on IP addresses, IP ranges, User-Agent strings, or suspicious request headers. Properly configured rules can help reduce unwanted traffic and improve server resource usage while protecting your website from abusive crawlers.