The Internet Is For Robots

Bot Traffic

A new report from web security company Incapsula shows that as much as 61% of web traffic is from bots, not real people. Frederic Lardinois unpacks it:

At first glance, this sounds like this means the number of nefarious attacks is up, but Incapsula actually notes that the bulk of growth in this number is due to what it calls “good bots.” Visits from certified agents from search engines and similar tools increased from 20 percent to 31 percent, for example. According to Incapsula, many search engines have lately increased their sampling rates. In addition, the SEO tools that try to help websites rank higher once they are crawled, also now often visit sites more often than ever before.

He adds an important caveat, that “Incapsula gathered this data by looking at stats from its own 20,000 customers” and that “companies that sign up for the kinds of security services the company offers may not be exactly representative of the Internet as a whole.” Still, Alexis Madrigal worries about the bots impact on the digital economy:

The point is: It’s so easy to build bots that do various things that they are overrunning the human traffic on the web. Now, to understand the human web, we have to reckon with the logic of the non-human web. It is, in part, shady traffic that allows ad networks and exchanges to flourish. And these automated ad buying platforms — while they do a lot of good, no doubt about it — also put pressure on other publishers to sell ads more cheaply. When they do that, there’s less money for content, and the content quality suffers. The ease of building bots, in other words, hurts what you read each and every day on the Internet.