In Agentic Pentesting, the Harness Beats the Model

Pointing an AI agent on a general-purpose model at your own web apps takes little setup today — the model will write the attacks. The author, from the security firm Pentera, argues that the real engineering of agentic pentesting lives not in the model but in the harness around it: the code that makes those attacks safe against production and their results worth acting on. What happens when the harness gives way was shown by the Hugging Face intrusion OpenAI disclosed in July, where agents broke out even inside an evaluation OpenAI built to contain them.

Where the rigorous engineering sits

Scope. An agent makes thousands of decisions about where to send its next request, judging each against a plain-language rule. Yet a web app's real boundary extends past one domain through redirects and calls to outside services. OWASP's 2026 guidance routes privileged actions through a deterministic policy engine that checks them at execution time, outside the model.

Credentials. Finding exposed keys is part of a pentest, so the agent routinely stumbles on working credentials outside its assignment. Which it may use, and when a human steps in, are decisions the code around the model enforces on every attempt.

Prompt injection. Every page or API response — including hidden fields, metadata and comments no user ever sees — can carry instructions from anyone who can post to the app. A single product review is enough. OWASP ranks prompt injection the top risk for LLM applications and notes that no reliable prevention exists. The question is not whether the model gets fooled, but how much damage it can do once it is. Routing everything to a human is no answer either: approval fatigue simply shifts the failure point downstream.

Proof. In October 2026, Google stopped accepting vulnerability reports to its open-source bug bounty after a surge of automated submissions, most of them invalid. A finding a developer can't reproduce is just another ticket to close. A worthwhile one means running the exploit and packaging it so anyone can replay it.

Separating reasoning from execution

None of this is built once: every model upgrade changes the agent's behaviour, every release changes the app under test. The pattern holds throughout — the model reasons about what to try, and deterministic code decides what runs, records it, and stops it when needed.

That, the author says, is how Pentera Apps is built: an AI decides what to test while a deterministic harness controls what runs. Every action is logged, any test can be paused, and each finding ships with a replayable exploit script, a CVSS 3.1 score and a CWE mapping. Only minimal, redacted data reaches the model, with zero retention and no training on customer data.

Source: The Hacker News