AI-Written Bug Bounty Reports Are Making Noise, and Not Always for Good Reasons

Every triager on every major bug bounty program has a version of the same story right now: a report comes in, formatted beautifully, full of confident technical language, citing a real CVE class, describing a plausible-sounding chain, and it falls apart the moment anyone actually tries to reproduce it. The finding doesn't exist. The endpoint behaves exactly as intended. The "vulnerability" is the model describing what a vulnerability of that type would generally look like, not what it actually found on the target in front of it. This is the AI-generated report problem, and it's reshaping how programs, including this one, have to think about triage.

How we got here

Large language models are genuinely useful for security work; a later piece on this blog covers exactly that. The problem isn't AI assistance itself, it's a specific failure mode: a researcher pastes a target URL or a snippet of an API response into a general-purpose chat model and asks it to "find vulnerabilities," and the model, which has no actual access to the target, no ability to send requests, and no ground truth about how the application really behaves, produces a plausible-sounding writeup anyway, because that's what generative text models do when asked a question they can't verify. The report reads like a real finding because it's built from the statistical shape of thousands of real findings the model was trained on. It just isn't grounded in anything that actually happened against the target.

Submit that unmodified, and you get what triage teams across the industry have started calling, only half-jokingly, "AI slop": a report with a title, a severity rating, a CVSS score, numbered reproduction steps, and a remediation section, describing a vulnerability that was never actually confirmed to exist.

What this looks like from the triager's side

A few patterns show up repeatedly:

Generic, templated technical detail that doesn't reference anything target-specific. "The application is vulnerable to SQL injection via the id parameter" with no actual payload, no actual response showing a database error or data leak, no evidence the parameter was even tested. Real findings quote real request and response bodies.

Confident severity language paired with zero reproducible evidence. Reports claiming "Critical, full account takeover" that, on inspection, describe a theoretical scenario rather than a demonstrated one, with no screenshots, no HTTP transcripts, nothing a triager can actually replay.

Reproduction steps that don't match the target's real behavior. Steps referencing UI elements, endpoints, or parameters that don't exist on the actual application, a strong signal the "testing" happened entirely inside a chat window rather than against the live target.

Fabricated or misattributed CVE references, cited with specific version numbers that don't correspond to anything the target is actually running.

None of this means every well-formatted, professionally written report is fake. It means well-formatted and professionally written are no longer signals of legitimacy on their own, which is a real shift, because for a long time, report quality and finding validity were loosely correlated. That correlation has weakened, and triage has had to adjust.

Why this actually matters, beyond wasted triager time

The cost isn't just the hours spent chasing down reports that go nowhere, real as that cost is. It's the effect on program trust and researcher relationships. Every hour a triage team spends on a fabricated report is an hour not spent on the genuine, high-quality submission sitting in the same queue, from a researcher who did the real work. Programs that get flooded with low-effort AI-generated noise sometimes respond by tightening acceptance criteria, slowing payout times, or raising the bar for what gets a full review, which affects everyone submitting, not just the researchers producing the noise.

There's also a subtler cost: it erodes the value of the report itself as a signal. Part of what makes bug bounty work is that a well-written, well-evidenced report is a strong prior that the researcher actually found something real. When that prior weakens, triagers have to verify more from scratch, which is slower for everyone, including the honest majority of submissions.

What separates a legitimate AI-assisted report from an AI-generated one

This distinction matters, and it's not about banning AI tools, which would be both unenforceable and counterproductive given how genuinely useful they are for the researchers actually doing the work (see the companion piece on this). It's about what the tool was used for:

• Legitimate use: AI helps you write clearer report prose, summarize a long reconnaissance session, suggest test cases to try next, or explain an unfamiliar framework's internals so you know where to look. The finding itself was discovered by actually interacting with the target.
• The problem pattern: AI is asked to invent the finding itself, based on a URL or a general description, with no actual testing against the live target to confirm the described behavior exists.

The test is simple and it's the same one triagers already apply: can you reproduce the reported behavior by following the exact steps against the exact target, right now? If yes, it doesn't matter how the report was written or what tools helped write it. If no, it doesn't matter how confident or technically fluent the writeup sounds.

What responsible programs, including this one, are doing about it

• Verification before severity assignment. Every report gets an actual reproduction attempt against the live target before any severity or bounty discussion happens, regardless of how the report reads on first pass.
• Requiring concrete evidence as a submission standard, not an optional nice-to-have: real request and response bodies, screenshots or recordings of the actual behavior, not paraphrased descriptions of what the behavior would be.
• Tracking pattern data on submission quality over time, which helps identify accounts submitting a high volume of unreproducible reports, a pattern that's usually more visible in aggregate than in any single report.
• Being direct with researchers about why a report was closed as not reproducible, rather than a generic rejection, since specific feedback is what actually helps someone improve, whether the underlying issue was AI misuse or simply insufficient testing.

What this means if you're submitting reports

Use AI tools. They're a genuine force multiplier for report writing, code review, and understanding unfamiliar systems, and pretending otherwise doesn't serve anyone. But the finding itself has to come from actually touching the target: real requests sent, real responses received, real behavior observed and documented. If you can't paste in an actual HTTP transcript proving the described behavior happened, you don't have a finding yet, you have a hypothesis, and hypotheses aren't what bug bounty programs pay for.

The researchers who consistently get paid, quickly, are the ones whose reports a triager can verify in minutes because the evidence is right there, target-specific and unambiguous. That was true before AI-assisted workflows became widespread, and it's still exactly the bar now.