The Security Bill for AI Written Code
Across 576,000 code samples, researchers found that 19.7 percent of the software libraries AI tools recommended did not exist. The models invented them. Worse, 43 percent of those invented names came back consistently across different prompts, which means they are predictable. Attackers have already noticed, and the attack has a name: slopsquatting.
You register the package the model keeps hallucinating. You put something nasty inside it. Then you wait, because sooner or later a build pipeline somewhere installs it without a human ever reading the name.
Start with the base rate
Before the exotic attacks, there is the ordinary problem. Analysis of AI coding tools puts their output at insecure roughly 45 percent of the time. Not broken, not failing tests. Insecure. The categories are the familiar OWASP set: weak input validation, SQL injection, hardcoded secrets, unsafe direct object references.
This is not a scandal about any one model. It is what you get when a system trained on public code reproduces the security posture of public code. The average snippet on the internet was written to demonstrate an idea, not to survive a hostile user. The model learned the average.
The model is not writing bad code. It is writing average code, at a rate no reviewer can keep up with.
Three failure modes worth knowing by name
Most of the damage clusters into three shapes. None of them look like a bug when you glance at the diff, which is exactly the problem.
01 · Hallucinated dependencies
One in five recommended packages is fictional, and the fictions repeat. An attacker who registers the common ones is not guessing, they are sitting on a known path. Your defence is a dependency scanner that blocks the build, plus a policy that no new package enters the tree without a named person approving it.
02 · The config file is now an attack surface
Agent instruction files sitting in the repository tell the agent what it may do. Anyone who can write to that file can rewrite the agent's authority. The same applies to a compromised tool server: a malicious connector can turn a helpful agent into remote code execution or a quiet data exfiltration route.
03 · Silent gray errors
The nastiest category. Studies of multi-agent systems found that 75.3 percent of failures come from a semantic breakdown when the planning agent hands work to the coding agent, and 75.17 percent of those failures compile cleanly, pass superficial checks, and then execute the wrong business logic in production. No alarm goes off. The invoice is just wrong.
That last one is the reason "the tests pass" has stopped being reassuring. A test written from the same instruction as the code does not check the code. It checks that the code agrees with itself.
What actually holds
The defences that work are unglamorous and mostly structural. Four of them do most of the job.
| Control | What it stops |
|---|---|
| Blocking scanners, not advisory ones | Hallucinated and vulnerable dependencies, leaked secrets, misconfigured infrastructure. At this volume, a warning nobody has time to read is the same as no check. |
| Builder and validator kept separate | Self-agreement. The agent that reviews the work must be a different agent with a different brief, judging against the original acceptance criteria rather than against the implementation. |
| Testing the outcome, not the diff | Silent gray errors. Running the finished behaviour against the original objective yields a measured 15.6 point absolute improvement in task success and catches logic that passes every unit test. |
| Least privilege for the agent | Excessive agency. Scope what it can reach, log every tool call, and require a human signature on anything that touches money, customer data or production infrastructure. |
Questions for whoever is building your software
You do not have to become a security engineer to tell the difference between a team that has handled this and a team that has not. Ask these and listen for specifics.
- Which checks in your pipeline can fail a release, and when did one last do it?
- How does a new third party package get into our project, and who approves it?
- If an AI agent wrote a change, what independently verified that it does the right thing?
- What can your agents reach in production, and where is that written down?
- Who reviewed the last change that touched authentication or payments?
A team that has done the work answers with names, tools and dates. A team that has not will tell you their AI is very good.
None of this is an argument against using AI
We build with these tools daily and we are not giving them up. The productivity is real. What changed is that the cost of a mistake now arrives at machine speed, and the old safety net, a senior engineer reading every line, does not scale to the volume being produced.
So the net has to move. Fewer eyes on individual lines, far more rigour at the boundaries: what enters the dependency tree, what the agent is permitted to touch, and what proves the finished thing actually does what was asked. That is the same discipline we described in harness engineering, pointed at security instead of reliability.
If you are running software that was written quickly and you are not sure what is underneath it, a dependency audit and a permissions review are a short piece of work with a large downside if skipped. Send us the stack and we will tell you what we would check first.
AI generated code securityslopsquattingpackage hallucinationAI supply chain securitysecure AI developmentOWASPMCP securityAI code vulnerabilitiesAI security Cyprusagent least privilege
Let's build
something real.
Tell us about the workflow, the bottleneck, or the channel that's bleeding margin. We'll show you how we'd ship the fix.