The bot traffic dilemma

Usually bots are bad, which is why some CSOs ensure their organizations’ Web sites block them.

There is a risk, a study by a security vendor has discovered, although not in the way most security pros think.

Blocking bots also means blocking Googlebots, which eagerly crawl the Web to come up with an accurate way of indexing content. So IT professionals set firewall rules that allow Googlebots on their sites. However, hackers know that, and have crafted malicious bots that mask themselves as Googlebots to find ways to exploit browser, server and other vulnerabilities.

The dilemma, as outlined by Igal Zeifman, product evangelist at Incapsula, which offers cloud-based Web site security, is that if you block bots to protect against fake bots, you also block legitimate Googlebots.

“It’s a no good options scenario,” he said in an interview.

INSIDE Incapsula infograf

Incapsula released a study of Googlebot activity Thursday which covered 400 million searches and visits, to 2.9 billion pages . It showed that on average Googlebots drop by a Web site 187 times a day. But for every 24 visits a Web site will also be hit by a fake version.

Over 23 per cent of those are used for denial of service (DDoS) attacks. Almost 11 per cent are malicious screen scrapers, spammers and scanners.

Bad bots come mainly from the U.S. (25 per cent), plus China (15.6 per cent), Turkey (14.7), Brazil (13.5) and India (8.4 per cent).

One problem, Zeifman said, is that legitimate Googlebots can come from the U.S., Britain, China, Denmark, Belgium and France, so blocking bots from those countries doesn’t help.

“I don’t think that many realize that not all Googlebots are who they say they are,” he says. “There are few services that show inside bot traffic,” he said.

Would you recommend this article?


Thanks for taking the time to let us know what you think of this article!
We'd love to hear your opinion about this or any other story you read in our publication.

Jim Love, Chief Content Officer, IT World Canada

Featured Download

Howard Solomon
Howard Solomon
Currently a freelance writer, I'm the former editor of and Computing Canada. An IT journalist since 1997, I've written for several of ITWC's sister publications including and Computer Dealer News. Before that I was a staff reporter at the Calgary Herald and the Brampton (Ont.) Daily Times. I can be reached at hsolomon [@]

Related Tech News

Tech Jobs

Our experienced team of journalists and bloggers bring you engaging in-depth interviews, videos and content targeted to IT professionals and line-of-business executives.

Tech Companies Hiring Right Now