A security agent should be measured by the work it completes and the evidence another person can inspect. A generated finding, a plausible repair and a verified outcome answer different questions. This report starts with Cefense’s recorded development evaluation and keeps those questions separate.
Cefense / Recorded development evaluation112 known cases.
112 known cases.
Four views of the same run.
56 vulnerable cases and 56 safe controls. Recorded 2026-10-23 UTC.
Model findings retained after source-context verification.
Cefense candidateCurrent candidate in this record
Vulnerable detected
25 / 56
Safe cases flagged
0 / 56
Recorded baselineSame labelled cases
Vulnerable detected
39 / 56
Safe cases flagged
1 / 56
Recall on these cases44.6%
Vulnerable cases missed31
Failed / not run0 / 0
This is a familiar, label-known development set. It does not estimate accuracy on your repository or rank Cefense against another vendor. Model: deepseek-v4-flash.
The candidate detects 14 fewer vulnerable cases than this baseline. It also flags 1 fewer safe cases. Single stochastic runs cannot isolate the effect of individual implementation changes.
Explore a 20-30% research target
Planning scenario / Not a measured result
The target below is derived from the selected Cefense candidate rate. It is excluded from the recorded results and from every comparison.
Recorded recall 44.6%Unmeasured target 55.8%
A relative 25% increase is 11.2 percentage points for this selection. No experiment has established this target.
One run, four views
The paired record contains 112 author-labelled cases: 56 vulnerable examples and 56 safe controls. The labels and source were familiar during development. The model configuration recorded for this candidate is deepseek-v4-flash. This is a development result, not an unseen customer-repository accuracy estimate.
Native detection uses the deterministic rules without model output. Mapped model detection checks the model’s finding against the declared CWE and sink location. Source-context verification then decides which mapped findings to retain. The combined result takes the union of native and verified model findings. Keeping the lanes separate shows which part of the system contributed an outcome.
Read coverage and noise together
In the verified-model lane, the candidate detected 25 of 56 vulnerable cases and flagged 0 of 56 safe controls. The recorded baseline detected 39 vulnerable cases and flagged 1 of 56 safe controls. The candidate reduced the safe-case flag count and missed more known vulnerabilities. The comparison keeps both changes visible.
The native-and-verified-model union has a different result from the verified model alone. That difference is the reason to inspect each lane. A product-level claim cannot be inferred from whichever lane has the most appealing number. Single stochastic runs also cannot isolate the effect of an individual prompt, grouping or verifier change.
A result needs a workflow
Detection is one handoff in a larger process. A team still needs to understand the behavior, inspect the proposed change, exercise the relevant checks and decide who can approve a release. The following diagram explains those handoffs. It does not assert that a customer scan completed them.
The agent workflowFrom a question
From a question
to a reviewable change.
Six handoffs explain the work. Choose a stage to inspect its purpose and output.
Stage 01 / A bounded assignment
Which repository, revision and actions are in bounds?
An agent needs a concrete job and the environment to do it. Repository access, test permissions and the intended result define the work.
- Repository and revision
- Permitted tools
- Review boundaries
This is a workflow explanation. It is not a replay of a completed customer scan or a measured product comparison.
Choose the test after the question
An exploit-reproduction benchmark asks whether a system can trigger a known behavior. A patch benchmark asks whether a change satisfies its grading conditions. A detection corpus measures matching against its labels. These protocols have different tasks and denominators, so their percentages cannot be combined into one leaderboard.
Workflow stages within benchmark types
The archived stage map is a way to choose a test, not a measurement of Cefense on those tests. Inspect which stage each protocol exercises before comparing results.
Archived research. Cefense profiles describe design intent, not a measured run.
The diagram loads as you approach it.
Explore all 50 benchmark profiles and their stage coverage
The benchmark libraryDifferent tests.
Different tests.
Different questions.
All 50 benchmark profiles from the November 5 research archive. Use the stage map to understand which part of a security workflow each test is designed to examine.
Cefense
Our recorded development results appear first in the measured report. The library below describes external protocols; a Cefense result on one protocol cannot be transferred to another.
Open Cefense’s measured evidence50 benchmarks
The descriptions and stage marks are the archived author’s assessments. Product performance is not scored here. Code and benchmark availability may have changed since the snapshot.
Source assessment: Closest thing to a full find-prove-fix benchmark that exists
Scope to inspect: Memory-safety C/C++ only; disclosed, already-fixed bugs
01FindIncluded02UnderstandIncluded03PrioritizeUncredited04Propose a fixIncluded05DeployUncredited06RetestIncluded07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: Most rigorous one-time test of autonomous find-and-patch
Scope to inspect: A competition, not a rerunnable leaderboard
01FindIncluded02UnderstandIncluded03PrioritizeUncredited04Propose a fixIncluded05DeployUncredited06RetestIncluded07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: Best web-app lifecycle benchmark; puts dollar values on results
Scope to inspect: Small; detection rates near the floor make ranking noisy
01FindIncluded02UnderstandIncluded03PrioritizeUncredited04Propose a fixIncluded05DeployUncredited06RetestIncluded07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: Scalable and real; good for tracking patch skill
Scope to inspect: Patch passes if it compiles and the PoC stops crashing
01FindUncredited02UnderstandIncluded03PrioritizeUncredited04Propose a fixIncluded05DeployUncredited06RetestIncluded07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: Best design against memorized fixes
Scope to inspect: 22 tasks is too few to separate close systems
01FindIncluded02UnderstandIncluded03PrioritizeUncredited04Propose a fixIncluded05DeployUncredited06RetestIncluded07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: Strictest patch grading published so far
Scope to inspect: Patching only
01FindUncredited02UnderstandIncluded03PrioritizeUncredited04Propose a fixIncluded05DeployUncredited06RetestIncluded07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: The most-cited AI cyber leaderboard
Scope to inspect: Top agent systems now report ~99%; it has little headroom left
01FindUncredited02UnderstandIncluded03PrioritizeUncredited04Propose a fixUncredited05DeployUncredited06RetestUncredited07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: Best independent test for EDR/XDR
Scope to inspect: No ranking; Microsoft, Palo Alto and SentinelOne skipped 2025
01FindIncluded02UnderstandIncluded03PrioritizeUncredited04Propose a fixUncredited05DeployUncredited06RetestUncredited07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: Strong realism for offensive web testing
Scope to inspect: Offense only; nothing about fixing
01FindIncluded02UnderstandIncluded03PrioritizeUncredited04Propose a fixUncredited05DeployUncredited06RetestUncredited07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Source assessment: Good functional-equivalence checking for patches
Scope to inspect: Fuzzing crashes in C/C++ only
01FindUncredited02UnderstandIncluded03PrioritizeUncredited04Propose a fixIncluded05DeployUncredited06RetestIncluded07Retain evidenceUncredited
Cefense on this exact protocol: no comparable result published in this record.
Targets are questions for the next experiment
The chart’s target explorer shows what a 20-30% relative improvement would mean for the selected candidate rate. That target is unmeasured, is never mixed into the result bars, and is capped when a request exceeds the metric ceiling. A target only becomes a result after a declared experiment produces its evidence.
Keep the complete record
The full benchmark destination retains earlier trials, their cost accounting, familiar-case regressions, fixture measurements, raw records and incomplete work. None of those records has been replaced by this visual summary. Read Cefense’s complete benchmark record for the protocols, case transitions and downloadable evidence.
The current record does not establish a cross-vendor performance ranking, hosted latency, accuracy on unseen customer code, or general patch correctness. A fair comparison needs the same tasks, repository revisions, tools, budgets, grading and retained failures for every participant.
