What it found on its own that pleased me
The clearest win for vulnerability_agent_ai was a boring one. Someone opened a PR against a small Python service and added a helper that read a user id off the query string and dropped it straight into a SQLite query with string concatenation. The kind of thing every scanner in the world should catch. Bandit catches it. Semgrep catches it. My agent caught it too. That is not the impressive part.
The impressive part is what it did after it caught it. The Code Analyzer flagged the line and marked it SQL Injection with High severity. The Vulnerability Classifier then picked up the finding, tagged it as CWE-89, mapped it to OWASP A03:2021, and pinned a CVSS score of 8.6. The CVE Verifier then went and actually opened NVD in a headless browser to check that the CWE number I was claiming existed, that the CVSS vector was plausible for a query-string SQL injection, and that the OWASP category was the one that actually covers injection. It came back with a confidence of 0.91 and a "should_publish: true". The Secure Refactor Agent then produced a parameterised query using ? placeholders and generated a unified diff patch against the file. The Compliance Reporter Agent then wrote the PR comment.
None of the four LLM calls in that chain were doing anything I could not do myself. But the fact that they ran in parallel across four vulnerabilities in the same PR, and that the verifier caught one instance where the classifier had hallucinated a CWE that does not exist (it made up CWE-943 for a case that was really CWE-78), was the moment I stopped treating this as a toy.
Where it did well
Injection classes were the sweet spot. SQL injection through string concatenation, command injection via os.system with unsanitised input, unsafe subprocess.check_output(shell=True), insecure pickle.loads on network data, eval of untrusted strings, path traversal via open(user_input) without normalisation. These all have a stable syntactic signature. The LLM did not really need to reason about them; it just needed to recognise the shape. In those cases, the four-agent pipeline gave me a scanner that produced a CWE, an OWASP tag, a CVSS score, a patch, and a PR comment in one shot, and the verifier was correct to publish about 92 percent of the time on my private benchmark of eighty-odd Python samples.
Hardcoded credentials worked too. AWS keys with the AKIA prefix, sk-proj- prefixes for OpenAI keys, database URLs with an inline password, Bearer tokens in header defaults. The regex-plus-LLM combo caught these reliably. Weak crypto (MD5 and SHA1 for password hashing, hardcoded IVs, ECB mode) was also easy for the model.
The dependency graph analyser was a piece I did not expect to work as well as it did. When a PR changed one file, the graph builder walked imports and pulled in every file that referenced the changed one, plus every file the changed one imported. The scanner then ran across that whole set. That let me catch a case where a helper was fine in isolation, but a caller two files away was passing user input into it without sanitising, so the helper became the sink for an injection. That is not something a per-file scanner would ever notice.
Where it fell short
Business logic bugs. The agent never once caught a real access-control mistake. On a Flask endpoint that read session["user_id"] but then took a target_user_id from the request body and edited that user's profile without checking they were the same, the scanner said "no vulnerabilities detected". It did not know what an authorisation model was. It could see the code. It could not see the intent. This is not a bug in my agent so much as the ceiling of syntactic pattern matching. OWASP A01 (Broken Access Control) is the number one item on the top ten and my scanner was blind to almost all of it.
Race conditions and TOCTOU bugs. Same problem. The agent read code line by line. It did not model time. If a check on line 12 was invalidated by an action on line 30 in a concurrent request, the agent had no way to see that. I fed it a classic double-spend pattern with if balance >= amount: balance -= amount and it flagged nothing.
Cryptographic misuse beyond the trivial cases. It caught MD5. It did not catch a case where I used AES-CBC with a predictable IV derived from the message id. That is a real bug and a bad one, and the scanner had no template for it. Anything that needed the model to reason about the actual crypto guarantees rather than pattern match on function names was outside its range.
Third-party dependency vulnerabilities. This was a strange gap because I had wired up an intelligence agent that pulled recent CVEs from NVD and a research feed. But the matching logic was crude. It checked whether any string in a CVE's affected_products list appeared as a substring in the code. That produced a river of false positives when the affected product was something with a common name like "requests" or "http", and it missed real matches when the code imported a package with a slightly different spelling than the CVE recorded. A real dep-vuln scan needs to read requirements.txt and pyproject.toml, resolve versions, and match against a version range. Mine did not do that. Snyk and Dependabot do. That is the honest gap.
Golang coverage. I wrote a Golang scanner. It shipped. It found the easy stuff (SQL injection via db.Query(fmt.Sprintf(...)), exec.Command with user input). But Go has whole classes of bug that the Python-focused prompts did not translate to well: unclosed response bodies, nil-map writes, and the subtler goroutine and context-cancellation patterns. I had regex-level patterns for goroutine leaks and TOCTOU races on paper, but they fired on shape (a go func(){ without a nearby defer wg.Done) and missed the actual concurrency bugs I fed it. Unclosed resp.Body and nil-map writes it had no template for at all.
The verifier is what saved the pipeline
The single biggest architectural choice that worked was making the CVE Verifier a separate step. The Classifier hallucinated CWE numbers about 8 percent of the time on my benchmark. Not by a lot; it would say CWE-89 when the real answer was CWE-564 (SQL injection through Hibernate). Or it would give a CVSS score off by two full points. The verifier ran a headless browser via browser-use against NVD and MITRE, pulled the actual CWE title, and either confirmed or rejected the classification. When it rejected, should_publish came back false and the finding got filtered out of the PR comment.
Without the verifier, the pipeline would have published wrong CWE numbers under my name in PR comments. That would have been worse than not running the scanner at all, because engineers stop trusting a tool the moment they catch it in one confident lie. With the verifier, the numbers that actually made it into the PR comment were the ones I would have signed off on myself.
The boundary between agent and human
Reading back through the code and the ninety-odd scan results I saved, there is a pattern to where the human still had to step in.
The agent could do the recognition part. It could pattern match a vulnerable snippet, tag it, score it, and produce a fix that compiles. On the twelve or so injection-class vulnerabilities in my benchmark, my patch was accepted with no changes by whoever was reviewing the PR about eight times out of ten. That is real. That saved real time.
The agent could not do the intent part. Nothing in the pipeline asked "what is this code for". A user_id from the request body being passed straight to a database update was fine if the endpoint was POST /admin/impersonate and the caller was an admin. It was a critical access-control bug if the endpoint was POST /profile/update. Same code, different intent, opposite verdict. The scanner treated them identically and had no vocabulary to distinguish them.
The agent also could not decide whether to fix or wait. On one PR the scanner found a High severity finding in a test fixture. Technically correct: the test file did indeed hardcode a password. But the password was the test-only credential for a mocked database, and fixing it would have made the test more confusing without making anything safer. A human read that in three seconds. The agent tried to generate a patch that pulled the value from an environment variable, and that patch would have broken the test on any machine that did not have the env var set. I ended up adding an .gitignore-style allowlist for test fixtures, which is another way of saying the agent needed a human to draw the boundary of where it was allowed to swing.
So the shape of the boundary was this. Recognition and mechanical fix generation: the agent could do it. Deciding whether a finding was in scope, whether the fix belonged in this PR, and whether the code actually did what the code appeared to do: I had to do that. Somewhere around eighty percent of the work by count, twenty percent of the work by weight.
What I would build differently
If I picked this back up tomorrow, three things would change. First, I would rip out the substring-matching intelligence agent and replace it with a real version-resolving requirements.txt scan against the OSV database. That is a well-defined problem with a well-defined data source; there is no reason to guess at it. Second, I would add a small "endpoint intent" annotation that a developer could drop on a route (# @intent: user-modifies-own-profile) and teach the classifier to compare the code path against the intent. That is the smallest step towards catching the access-control class of bug without pretending the LLM can infer intent from scratch. Third, I would delete the emoji-decorated PR comments and replace them with plain text. Reviewers took the tool less seriously the moment they saw a fire emoji next to a finding.
The scanner works. It found real bugs, it filtered its own hallucinations, and it wrote patches I mostly kept. It also missed a whole floor of the OWASP top ten, and I have no honest way to spin that. That is the retrospective.