Data Poisoning
Data poisoning refers to attacks in which someone deliberately smuggles false or manipulated examples into an AI's learning materials. The program learns the intended errors from this and later behaves exactly as the attacker planned.
Computer programs called AI learn from examples. They are shown millions of texts or images, and from these they derive patterns. This collection of examples is called training data. Data poisoning means that someone deliberately corrupts such examples before the program sees them. The attacker thus does not attack the finished program, but its learning material. The result: the program learns something false – and nobody notices right away. The German term for this is Datenvergiftung.
Why this matters
Large language models are trained to a considerable extent on texts from the open internet. Anyone can write there. This means that, in principle, anyone can have a say in what a model learns. This makes training data a vulnerability that classic security technology does not cover. A firewall doesn’t help against a Wikipedia entry with fabricated numbers.
Particularly unpleasant is the fact that it happens after the fact. A poisoned model cannot simply be repaired. Usually the training has to be repeated, and that costs millions. That is why regulators now treat data poisoning as a serious risk. The EU’s legal framework for AI requires providers to check the origin of their data.
How it works
Two basic forms can be distinguished. In the first, the attacker wants to degrade the model in general. He scatters nonsense in large quantities until quality declines. This is crude and is often noticed.
The second form is more targeted and more dangerous. The attacker builds in a hidden entry point, known in the jargon as a backdoor. To do this, he links an inconspicuous signal with a desired behavior. One example: in a hundred training images, a small yellow sticker is stuck on a stop sign, and all of them are labeled “Speed 80”. The model learns the rule: yellow sticker means speed 80. In all normal tests, it behaves flawlessly. Only when the attacker applies the sticker does the trap spring shut.
What is astonishing is how little is needed for this. Studies by AI companies suggest that a few hundred manipulated documents can be enough. And this regardless of how large the model is overall. A larger model is thus not automatically better protected.
A related case concerns systems that load additional documents during operation, for example from a company database. Anyone who smuggles a manipulated document in there changes the answers immediately. The model itself remains untouched in the process.
Where the term is encountered
Two sides of the issue appear in the news. On one hand, security reports: researchers regularly show that expired internet addresses linked in well-known datasets can be bought up. Whoever owns the address determines the content.
On the other hand, the same technique is used as a protective measure. Tools like Nightshade alter artists' images minimally. To the human eye everything stays the same, but for a model the image becomes misleading. Anyone who uses the works for training without permission thereby damages their own product. Attack and self-defense are technically the same thing here.
For investors the term is relevant because data quality has become a competitive factor. Providers now advertise vetted and licensed data sources instead of pure internet collections. Clean learning material is expensive – but hard to poison.