The question
How do you tell whether a security-research system is getting better at finding bugs, or just getting better at sounding convincing?
More alerts and higher confidence scores don’t answer that question. I want to measure whether the results hold up when someone checks them.
What to measure
A useful benchmark needs both vulnerable code and code with no known bugs. It also needs to record the exact code version and build environment so results can be repeated. False alarms need to count against the system; otherwise, guessing at everything can look like progress.
I’m thinking about this in three stages:
- Discovery — does the system describe a specific bug that can be tested?
- Reproduction — can a test demonstrate it in an isolated environment?
- Verification — can an independent check repeat the result without relying on the original research conversation?
What remains open
Known vulnerabilities are a useful starting point. The harder question is whether lessons from them help the system investigate code it hasn’t seen before. Remembering a disclosed bug and discovering a new one are different achievements.