Collinear AI releases CWE-bench: a held-out test for coding agents’ defensive cybersecurity skills

# What CWE-bench is and why it matters

Collinear AI launched CWE-bench to measure whether frontier coding agents can defend real software by finding and repairing vulnerabilities. The benchmark is held-out: tasks use known open-source code and documented weaknesses, but the test cases are arranged so memorizing published fixes does not guarantee success. That design forces agents to reason about how a weakness can reappear in different parts of a codebase.

# Scope and structure

# How agents performed

Initial runs show the leading agent does not reach majority success. Specific results published by Collinear include:

• Fable 5 leads with a 47% pass@1 score at maximum reasoning.
• 18 out of 100 tasks are unsolved by every model tested.
• No agent evaluated successfully repairs more than half of the benchmark tasks.

Collinear also reports that performance varies significantly across weakness types. That variability points to concrete gaps in defensive software reasoning rather than a single broad failure mode.

# Example that illustrates the test's difficulty

# Performance versus cost

The benchmark includes cost measurements for agent rollouts. Models on the Pareto front balancing performance and rollout cost include Fable 5, Gemini 3.8 Flash Cyber, and GPT-5.6 Sol, all evaluated at high reasoning settings. That framing helps teams compare trade-offs between capability and compute or time cost.

# What CWE-bench offers builders and defenders

Collinear positions the benchmark as a practical measurement tool: teams can use the leaderboard and methodology to prioritize model improvements that target the weakest defensive skills.

# Where to find it

Collinear hosts a leaderboard and methodology details at cwe-bench.com and published a construction write-up on the Collinear blog.

# Bottom line

CWE-bench is a focused, taxonomic test of whether modern coding agents can repair real software vulnerabilities. Early results show substantive gaps: the best agent passes under half the tasks, many tasks are unsolved, and failures often reflect reasoning gaps rather than missing factual knowledge. The benchmark is intended to guide model builders toward concrete defensive improvements.