The Most Active and Friendliest
Affiliate Marketing Community Online!

“AdsEmpire”/  Direct Affiliate

Bot 'n Scraper Webserver Banning

Graybeard

Well-Known Member
How I ban bad bots, spy tools and web scrapers --

  1. you need root access in the server (VPS or above)
  2. you will need to edit the root files on the server in SSH (or a local terminal)
  3. you need rather obscure knowledge of networks, who to block, who NOT to block
  4. you need to scan your server logs CONTINUOUSLY finding new threats -- bad bots and scraping ips.
  5. you need to know how to find AS assignments and block entire network CIDR ranges
  6. you need to learn to use $ ipcalc
  7. you will need to spend too much time :( on defeating the bottomless pit of SEO scammers and thieves
Good luck with your spy tools and scrapers ...

$ curl -I 'https://www.<target-server>.com/'
curl: (52) Empty reply from server
^^^ 444= a server fart
_______________________________

In a Nginx domain configuration;

in the main
server{
Code:
#########
###access control
#allows

#first in order allow the ips you want here if
#they may be blocked in your .conf files
#example below
allow 34.35.67.102;

###deny conf

    # BLOCK SCAMMERS IP ADDRESSES
    include /etc/nginx/conf.d/blockips.conf;
    include /etc/nginx/conf.d/block_asns-hisec.conf;

#BAD BOTS

#map $http_user_agent $bad_bot
#map include in /etc/nginx/nginx.conf
#include /etc/nginx/conf.d/block_bots.conf;

if ($bad_bot) { return 444; }

if ($server_protocol = HTTP/1.0) { return 444; }

if ($http_user_agent = "") { return 444; }

##############
}
named referrers can be easily blocked also -- just need to know the names ...
This will never be 100% unfortunately. But you will see if the traffic you buy is junk or not ;)
 
Last edited:
Another bot trap is a relative linked JavaScript in the document <head>.
Most lame (scraper) bots will not request it (PhantomJS bots will ;))
 
banners
Back