A quick overview

Have you ever noticed traffic to your site from strange geographic locations or URL’s? Chances are if you use Google Analytics or Bing Webmaster tools that you have seen traffic coming from locations that your target audience wouldn’t normally be at, such as Russia or Ukraine. Strange?

On first inspection, it may be a little odd, however if you look a little further into it, you will start to see URL’s from sites such as the below:

 

In the above example we are using a clients site, you can see that most are the usual referrals you would get such as Facebook, LinkedIn and our own site, however you can also see 3 examples of spam sites. “Semalt.semalt.com”, “buttons-for-website.com” and “darodar.com”. These sites are spam sites that will generate lots of fake traffic which can actually harm your Search Engine Optimization (SEO) and could potentially slow down your site for legitimate users.

These spam bots are crawling your site multiple times a day, hindering your performance and generating traffic which when you look in your favourite analytics tool, isn’t going to be giving you an accurate representation of your website’s traffic. The fact that it could also be negatively affecting your search engine rankings is also a major factor.

So let’s get to it!

 How to block spam bots using .htaccess and Google Analytics

.htaccess

First of all, lets take a look at the .htaccess file. A word of forewarning, please be careful what you do in here, and always take a backup of the original just in case you need to revert.

This should be located in the root of your site most likely, however if you can’t see it, you can create one by simply creating a new file and naming it “.htaccess”. If you don’t have permissions to do this, you will need to speak to your host. You may also find the file exists, but may be blank or with very little information. This is also fine, everyone’s file will look different.

At the end of your file, let’s add in the following code:

#Spam bot block section
SetEnvIfNoCase Referer semalt.semalt.com spambot=yes

Order allow,deny
Allow from all
Deny from env=spambot

So let’s take a look, the first line (excluding heading) is to set environment variables based on attributes of the request without respect to the case.  So in this section, add all the referrers that you want to block. The second section is essentially allowing from all, however if the referrer is tagged  with the variable “spambot” then deny access to the page.

I’ve created the following list for you to use of some of the bots I have come across recently, however this is something you need to keep on top of as more popup from time to time. Look in your analytics to see if you can identify any more.

Here is the current list I am using, you can see where I’ve added them to the top section:

#Spam bot block section
SetEnvIfNoCase Referer semalt.semalt.com spambot=yes
SetEnvIfNoCase Referer blackhatworth.com spambot=yes
SetEnvIfNoCase Referer buttons-for-website.com spambot=yes
SetEnvIfNoCase Referer darodar.com spambot=yes
SetEnvIfNoCase Referer priceg.com spambot=yes
SetEnvIfNoCase Referer ilovevitaly.co spambot=yes
SetEnvIfNoCase Referer make-money-online.7makemoneyonline.com spambot=yes
SetEnvIfNoCase Referer hulfingtonpost.com spambot=yes
SetEnvIfNoCase Referer bestwebsitesawards.com spambot=yes
SetEnvIfNoCase Referer cenoval.ru spambot=yes

Order allow,deny
Allow from all
Deny from env=spambot

Add this code to your .htaccess file and it should block these annoyances! Now onto analytics in the next section.

Google Analytics

In this example, I am using Google Analytics to demonstrate how you can block these sites from appearing in your Analytics results. The same can be done with Bing Webmaster Tools however.

Open up Google Analytics and go into your web view for the site your working on. Then you need to identify the sites you need to block, you may already have these from the previous section however if you don’t, then go to Acquisition > All Traffic > Referrals and you should have something similar to the image below where you can identify spam sites.

Once you’ve taken a note of these, go to the Admin and under the Account section click All Filters and hit New Filter.

You will have a screen with a couple of options on it, fill out as demonstrated below. You will need to do this for each URL you have taken a note of.

And that’s it! You’ve successfully blocked these spam bots from accessing your sites and should have a much more accurate representation of true traffic on your analytics. Please note that it can take a few days for Google Analytics to show the changes, however your .htaccess changes should be immediate.

Thanks for reading and we hope that helps! Below I have created a list of ones to add that we have personally come across and will keep this updated with any new URL’s that crop up. Feel free to post any that you think you have below and I will add it to the master list!

Sites to Block:
blackhatworth.com
buttons-for-website.com
semalt.semalt.com
darodar.com
priceg.com
ilovevitaly.co
make-money-online.7makemoneyonline.com
hulfingtonpost.com
bestwebsitesawards.com
cenoval.ru
o-o-6-o-o.com

Last updated: 25/02/2015

Feel free to share if it helped you and leave any comments below!