Bot Traffic

Every time someone visits a website, their device sends a request to the computer on which the page is stored. Taken together, these requests are called traffic — that is, the traffic on a website. Part of this, however, does not come from humans but from programs that retrieve pages on their own. Such programs are called bots, and their visits are referred to as bot traffic. At first, the operator of the site simply sees that a request has arrived — whether a human or a machine is behind it has to be figured out first. Bot traffic itself is neither automatically harmful nor automatically harmless: it depends on which program is doing what.

Useful bots and harmful bots

Some bot traffic is desirable. Search engines send out programs that read pages and add them to their index. Without these so-called crawlers, a website wouldn’t even show up on Google. Price comparison portals, news feeds, and monitoring services that report when a server goes down also work with bots.

Other bots cause damage. They try out stolen passwords on login pages, buy up sought-after concert tickets within seconds, or copy a competitor’s entire product catalog. Others click on advertisements so that the advertiser has to pay, even though no human ever saw the ad. According to estimates, a double-digit percentage of all web traffic can be attributed to such malicious programs.

For businesses, this has direct financial consequences. Every request costs computing power, and anyone with very high traffic pays more to their hosting provider. Even more troubling are distorted figures: if an online shop believes an advertising campaign brought in 100,000 visitors, but half of them were machines, it will base its next decisions on false premises.

How to distinguish machines from humans

Every request to a website contains additional information, such as which browser and which operating system are being used. Well-behaved bots identify themselves honestly there, for example as “Googlebot.” They also generally comply with a file called robots.txt, in which a website operator notes which areas programs should not access. However, this is merely a request, not a technical barrier.

Malicious bots pretend to be a normal browser. That’s why one looks at their behavior instead. A human moves the mouse irregularly, scrolls, pauses, and takes seconds to read. A program, by contrast, retrieves 300 pages per minute at exactly even intervals. The origin of the request is also a clue: if thousands of requests come from a data center rather than from mobile or home internet connections, that is suspicious.

Countermeasures range from puzzles like CAPTCHAs to systems that simply slow down suspicious requests. None of this ever works perfectly. Block too strictly, and you lock out genuine customers — a familiar annoyance for people using an ad blocker or a VPN. So one is always just shifting the line between too much and too little protection.

AI crawlers as a new point of contention

For a few years now, there has been a new kind of bot: programs from AI companies that collect text and images from the web in order to train language models. Many publishers and forums see this as a problem. Their content gets scraped, later shows up in AI answers, and readers never visit the original site. As a result, more and more websites are blocking these crawlers in their robots.txt or demanding payment for access.

In business news, bot traffic comes up mainly in three contexts. First, with providers like Cloudflare or Akamai, whose business is filtering out unwanted access. Second, in disputes over ad fraud, when companies discover they have paid for machine-generated clicks. Third, in lawsuits between media companies and AI providers over the question of who is allowed to automatically collect content.

A common misconception is that bot traffic is the same as a hacker attack. Most of it is simply routine: search engines, security scanners, price-tracking robots. It only becomes noticeable when a server collapses under the load — that’s when people speak of a DDoS attack, in which thousands of computers deliberately overload a site.