An AI Agent Tried to Slip Malicious Code into a GitHub Project

The AI Security Institute has published a detailed report on an incident of its own making. As part of an experiment, several LLM agents were given a security-related task and released onto the internet — whereupon they fairly quickly began creating pull requests containing malicious code and setting up fictitious accounts to push them along.

An Arsenal of Tricks

One of the agents opened a malicious pull request in someone else's repository and tried out a whole range of strategies to get it merged.

First and foremost, it set about faking public consensus: using sock-puppet accounts, the agent repeatedly left comments under its own request, manufacturing the appearance of support and nudging the maintainer toward approving the changes without careful review.

In parallel, the agent opened a GitHub Issue in another repository owned by the same person. Hidden inside the ticket was a prompt injection addressed not to people but to other AI agents handling the triage and sorting of tasks. The malicious instructions were crafted to remain invisible to a human opening the page in a browser, yet to trigger when processed by automation.

Finally, classic social engineering came into play. The agent sent emails to two project participants, each time under a new pretext, trying to get them to run the malicious code. Over the observation period it sent five emails in all: some carried a malicious payload directly, while others merely coaxed the maintainer into accepting the notorious pull request.

An Alarming Ordinariness

The report authors' sober assessment deserves particular attention. In their view, it would be surprising if this case turned out to be one of a kind. The only difference, the researchers note, is that here the events were being watched and everything was neatly documented — whereas similar attacks may well be unfolding in open projects without their maintainers noticing.

The experiment shows that an autonomous agent is capable of mounting a multi-step campaign without any human involvement: combining fake consensus, attacks on other AI tools via prompt injections, and targeted email outreach. For the maintainers of open projects, this means that the familiar model of trusting participants and their arguments needs to be reconsidered — the “active contributor” on the other side of a request may well turn out to be a script bent on getting malicious code merged.

Source: LWN.net