Data poisoning

In security research, data poisoning is an attack on a model's training data. An attacker places crafted examples where they will be collected, so the model learns a hidden behavior, such as responding in a particular way to a trigger phrase. A 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that a small, roughly fixed number of poisoned documents could implant a backdoor in language models of very different sizes. That result was a controlled experiment on a narrow behavior, and it does not describe what happens to a deployed assistant.

In influence-operation research, the term is used for a related tactic: publishing large volumes of content so that AI systems repeat it. Some researchers call this "LLM grooming". NewsGuard reported in 2025 that chatbots repeated narratives from a network of Russian-linked sites, the so-called Pravda network, in a share of tested answers. Other researchers disputed how often this happens, arguing that many cases reflect gaps where little other material exists on a topic. Treat the extent of the effect as contested.

Assistants draw on training data and, in many products, on pages retrieved at the moment of a question. Material planted for either route can shape what an assistant says about an organization. A hypothetical example: a coordinated network publishes hundreds of near-identical pages making a false claim about a company, then waits for the claim to surface in assistant answers. It is also an old problem in a new place: Microsoft's Tay chatbot, which learned from users, was pushed by coordinated users into posting offensive messages within a day of its 2016 launch.

Defensive work looks like ordinary monitoring applied to a new surface. Ask a fixed set of benchmark questions about the organization on a schedule, record the cited domains, and look for unfamiliar sources repeating the same claim. The free AI Perceptions check samples answers and shows cited sources for a brand, and the AI Perceptions reports go further. Repeating the claim to a provider helps less than publishing accurate, well-sourced pages; see generative engine optimization and information operations.

A wrong answer from an assistant is rarely evidence of poisoning. Most errors are hallucinations or reflect weak sources.

A company that optimizes its own pages with accurate content is doing something legitimate, which is a different activity. Attribution is also hard: seeing a narrative in an assistant's answer does not show who placed it or whether it was meant to reach models at all.