Graybeard
Well-Known Member
How I ban bad bots, spy tools and web scrapers --
$ curl -I 'https://www.<target-server>.com/'
curl: (52) Empty reply from server
^^^ 444= a server fart
_______________________________
In a Nginx domain configuration;
in the main
server{
}
named referrers can be easily blocked also -- just need to know the names ...
This will never be 100% unfortunately. But you will see if the traffic you buy is junk or not
- you need root access in the server (VPS or above)
- you will need to edit the root files on the server in SSH (or a local terminal)
- you need rather obscure knowledge of networks, who to block, who NOT to block
- you need to scan your server logs CONTINUOUSLY finding new threats -- bad bots and scraping ips.
- you need to know how to find AS assignments and block entire network CIDR ranges
- you need to learn to use $ ipcalc
- you will need to spend too much time on defeating the bottomless pit of SEO scammers and thieves
$ curl -I 'https://www.<target-server>.com/'
curl: (52) Empty reply from server
^^^ 444= a server fart
_______________________________
In a Nginx domain configuration;
in the main
server{
Code:
#########
###access control
#allows
#first in order allow the ips you want here if
#they may be blocked in your .conf files
#example below
allow 34.35.67.102;
###deny conf
# BLOCK SCAMMERS IP ADDRESSES
include /etc/nginx/conf.d/blockips.conf;
include /etc/nginx/conf.d/block_asns-hisec.conf;
#BAD BOTS
#map $http_user_agent $bad_bot
#map include in /etc/nginx/nginx.conf
#include /etc/nginx/conf.d/block_bots.conf;
if ($bad_bot) { return 444; }
if ($server_protocol = HTTP/1.0) { return 444; }
if ($http_user_agent = "") { return 444; }
##############named referrers can be easily blocked also -- just need to know the names ...
This will never be 100% unfortunately. But you will see if the traffic you buy is junk or not
Last edited:
