Threat Actors Are Poisoning the Well of AI Training Data

Data is at the core of AI. Developers want to make sure the well isn’t poisoned. 

Accenture’s recent State of Cybersecurity Resilience report found that companies’ top AI-threat concern is the poisoning of the data they use to train their AI. Threat actors have taken to data poisoning by contaminating upstream datasets to subtly shift model behavior, often targeting open-source or weakly curated datasets, said Dewank Pant, AI security researcher at Amazon. 

“When threat actors create malicious content by hijacking…these are known as inference-time poisoning or indirect prompt injection,” said Pant. 

So how can a training dataset become poisoned? Some tactics include publishing listicles on hacked sites and rebuilding expired domains that AI models trust and regularly cite. Other types of AI poisoning are expected to emerge as cybercriminals continue to get creative in their techniques, Pant said.  

These attacks can extend to agentic systems, too, said Pant, such as threat actors poisoning metadata to manipulate how large language models execute external actions. 

Hackers who breach a company may remain in its systems for long periods of time, even months, said Tony Anscombe, chief security evangelist at ESET. 

“During this time, they could potentially observe business operations and selectively manipulate data used to model systems in a way that gradually undermines AI performance,” Anscombe said. “By the time the impact becomes noticeable when AI output can no longer be trusted by the business, it may be too late to easily identify or reverse the changes.” 

Although uncommon, these sophisticated cyberattacks can impact AI-reliant workflows, including risks such as faulty, biased or even dangerous business decisions, Daniel Schiappa, chief product and services officer at Arctic Wolf. Attackers may even ransom in exchange for restoring the AI’s performance. 

But there are a few steps enterprises can take to mitigate these risks, said Schiappa. 

• The first is controlling prompts. Enterprises must know where their training data comes from and verify it. “Limit dependence on publicly scraped sources for sensitive or critical AI workflows,” said Schiappa. 
• The second is to use trusted pipelines. It’s important to leverage secure data pipelines with integrity checks and versioning, he added. This means checking where the data is stored and moves to and from. 
• The third is to flag anomalous patterns using new technologies. “These are worth exploring, especially for proprietary AI models,” Schiappa said. Fourth, enterprises should build human-in-the-loop checkpoints and transparency layers, especially for high-impact AI decisions. 

“It’s important for enterprises to treat AI poisoning like any other threat vector and integrate it into security operations and threat intelligence programs,” Schiappa said.