AI Code Packages Open Doors For Hackers, Study Finds

AI-generated code often includes fake software libraries, creating new opportunities for hackers to exploit supply chains and compromise users across development platforms.

Research indicates that AI tool-generated code creates substantial security vulnerabilities which threaten the software supply chain. The research, first reported by Ars Technica, indicated that large language models (LLMs) which operate similarly to ChatGPT systems generate fictional code dependencies which hackers can potentially use for malicious purposes.

Ars reports that the researchers evaluated 16 widely used AI models through the generation of 576,000 code samples. The analysis revealed that 440,000 package references were hallucinated because they pointed to non-existent code libraries.

The existence of these fabricated dependencies creates a significant security risk. Ars reports that attackers can identify repeated AI suggestions of package names to upload malicious packages with those names. The attacker gains control of a developer’s system when they unknowingly install the malicious code.

“Once the attacker publishes a package under the hallucinated name, containing some malicious code, they rely on the model suggesting that name to unsuspecting users,” explained Joseph Spracklen, a Ph.D. student at the University of Texas at San Antonio and lead researcher, as reported by Ars.

“If a user trusts the LLM’s output and installs the package without carefully verifying it, the attacker’s payload, hidden in the malicious package, would be executed on the user’s system,” Spracklen added.

The attack method tricks software into selecting a dangerous package version instead of the intended correct version, as reported by Ars. The dependency confusion attack affected major technology companies, including Apple, Microsoft, and Tesla, during previous testing.

The researchers discovered that open-source models, like CodeLlama, generated more hallucinated packages than commercial models did. The open models generated false code references at a rate of 22%, while commercial models produced hallucinations at 5% or less. The JavaScript programming language experienced more hallucinations than Python because it operates within a larger and more complex code ecosystem.

According to the study, these are not just one-off mistakes. The study reported that many fake packages appeared repeatedly in different tests, which makes them more dangerous because they can be targeted more easily by attackers.

Ars explains that attackers could exploit repeated fake package names by uploading malware under those names, hoping developers unknowingly install them.