Massive *badBots* List

[Graybeard]

I found this on a referral website on Google News .> US coronavirus economic relief package - Schumer says vote could come on Sunday
I extracted the 'names' they used to a csv list the made a DOS *.txt file it's attached
the LINUX/UNIX file is /badbotlist.csv.txt
There are 2797 names badbotlist.csv
no guarantees but I looked and recognized many names I am currently blocking
USE WITH CAUTION
The script in the HTML source crashed my text editor the names are in one very long line
opens with Kate (LINUX) better edited in nano

Good luck to the winners --he made google news search for reason that the scraper, spy and link check bots were banned --could be

It may be just his googletag

        <script>
        function checkIfUserBot() {
            let isUserBot = false;
            const blackBotsList = "the list"for (let i = 0; i < blackBotsList.length; i++) {the list
                if (window.navigator.userAgent.indexOf(blackBotsList[i]) != -1) {
                    isUserBot = true;
                    break;
                }
            }
            return isUserBot;
        }
    

   <script>
        const isUserBot = checkIfUserBot();
        if (!isUserBot) {
            var googletag = googletag || {};
            googletag.cmd = googletag.cmd || [];
... see source ...

</script>

 

[Honeybadger]

@Graybeard
Does bots matter when my traffic is organic?
There is filter in Google Analytics so I can see stats with and without bots
Also, what exactly are bots?
& do many bots mean extra server cost?

 

[Graybeard]

If I use a spy-tool (on that bot list) and scrape and duplicate your content --is that a good thing?
Bots cause minor overhead because most only request HTML and not images, JS, large CSS files.

 

[T J Tutor]

My campaigns, as well as AffiliateFix, have been getting hammered this past week by bots. Heading them off at the pass so far!

 

[Graybeard]

My campaigns, as well as AffiliateFix, have been getting hammered this past week by bots. Heading them off at the pass so far!

but they change their name and address and keep on coming ...

 

[Aiden L]

The conditions are probably still there?

More spam unfortunately @azgold

 

[GriD]

More spam unfortunately @azgold

he has been spamming the forum...
Looks like I can't report this post right now...
View attachment 18980
@azgold @T J Tutor If we are held back from reporting the posts, shouldn't we have the option to report someone as a spammer without having to post off-topic ourselves?

 

[azgold]

If we are held back from reporting the posts, shouldn't we have the option to report someone as a spammer without having to post off-topic ourselves?

Yes, you should have that option.

I can see three reports from you. Are you saying that there were more and it wouldn't let you?

 

[GriD]

Are you saying that there were more and it wouldn't let you?

Yes but I misread the error message... I could have waited those 15 seconds

 

[Graybeard]

Yes but I misread the error message... I could have waited those 15 seconds

just report that user's message list URL along with that single post -- kill many babble-bots with one stone.

 

[songar]

but they change their name and address and keep on coming ...

If I understood you correctly, it is possible to limit participation, and highlight bots so that the system can block and distinguish between activity and bots, and even more to know the details and sources that send bots to the site?

 

[Graybeard]

The first thing I learned in security programing is only accept what you want. You cannot ban every eventuality --not possible to predict.

That said, I do ban known suspect networks and out of date browsers (user agents).

 

[MasterOfHearts]

I looked and recognized many names I am currently blocking
USE WITH CAUTION

Thank you so much for sharing this with us @Graybeard ! That is so very kind of you!

Ok, now how do I use this file? Import it in our tracker?

Thanks again!

 

[Graybeard]

in the server config
or in htaccess --use with Apache
but you should read your server access file and these 'should' be excluded by ad network traffic already.
The point of this list is to prevent bot to domain server scraping of content.

However most is Google headless chrome or selenium today --so technology has out dated that list really in the last 2year IMHO.
The plus is that some of the SEO scrapers still use those sigs --so that part still works.

Be careful the Mozilla/5.0 (compatible ; bots on that list include Google Bing and some other desirable bots too. Use with care!