Is this the right way to block user-agents I find on my logs?

Question

I want to block several agents that I have seen in my server logs, their exact names in the logs were, for example:

Python-urllib/2.6

Python-urllib/2.7

CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html)

I've seen this example, but not sure how to implement it, should it be exactly as they appeared:

RewriteEngine On 
RewriteCond %{HTTP_USER_AGENT} ^Python-urllib/2.6 [OR] 
RewriteCond %{HTTP_USER_AGENT} ^Python-urllib/2.7 [OR] 
RewriteCond %{HTTP_USER_AGENT} ^CATExplorador/1.0beta (sistemes at domini dot cat; http://domini.cat/catexplorador.html) [OR] 
RewriteRule ^.* - [F,L]

Or would this be good as well:

RewriteEngine On 
RewriteCond %{HTTP_USER_AGENT} ^python [OR] 
RewriteCond %{HTTP_USER_AGENT} ^catexplorador [OR] 
RewriteRule ^.* - [F,L]

Questions for example:

1. If I want to block all versions of this "python" (or all agents containing that string), would the second code work?

2. are the names case-sensitive? (catexplorador vs CATExplorador vs CaTExplorador)

3. Are these the right flags?

4. how to block "blank" user agents? (is it recommended?)

Answer 1

To block all agents starting with python OR catexplorador and making it case insensitive you can use:

RewriteEngine On 

RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteCond %{HTTP_USER_AGENT} ^(python|catexplorador) [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule ^.* - [F,L]

Answer 2

One option to block specific user agents is to use this tutorial.

BrowserMatchNoCase python bad_bot
BrowserMatchNoCase catexplorador bad_bot
Order Deny,Allow
Deny from env=bad_bot

The BrowserMatchNoCase allows case insensitive matches against the user agent.

A second option is to use the SetEnvIfNoCase command

SetEnvIfNoCase Referer "^$" bad_user
SetEnvIfNoCase User-Agent "python" bad_user
SetEnvIfNoCase User-Agent "catexplorador" bad_user
SetEnvIfNoCase User-Agent "^$" bad_user
Order Deny,Allow
Deny from env=bad_bot

This example blocks empty referrers (WARNING! NOT RECOMMENDED) and user agents that contain either python or catexplorador. You don't want to block empty referrers because they can be removed automatically by browsers. It also blocks empty user agents using the last line. Browsers can also do this automatically, depending on how they are configured.

Side note: User agents can be spoofed and changed at will and this is not guaranteed to block anything if the user changes their user agent.

The BrowserMatch is a special cases of the SetEnvIf directive that sets environment variables conditional on the User-Agent HTTP request header. The BrowserMatch can only check the user agent, while the SetEnvIf can check other parameters.