clauding.de
DE
• 2 min read AI-generated

OpenAI Audits SWE-Bench Pro — and Finds Nearly a Third of Tasks Broken

Copy article as Markdown

First OpenAI told the community to switch to SWE-Bench Pro. Now it's withdrawing that recommendation: up to 34% of tasks are defective. A wake-up call for anyone who takes coding benchmarks at face value.

OpenAI Benchmarks Coding Research
Featured image for "OpenAI Audits SWE-Bench Pro — and Finds Nearly a Third of Tasks Broken"

Coding benchmarks are the currency the AI industry uses to compare its models. That’s exactly why OpenAI’s new audit is so awkward — and so important.

From recommendation to withdrawal

After flaws were found in SWE-bench Verified, OpenAI had recommended the community move to the newer SWE-Bench Pro benchmark. Now the company has turned the same critical lens on SWE-Bench Pro — and withdrawn that recommendation.

An automated analysis pipeline flagged 286 of the 731 tasks in the public set as potentially problematic. A follow-up review — combining investigative Codex agents with five independent human engineers per task — confirmed that 200 to 249 tasks are actually defective. Depending on how you count, that’s 27.4% to 34.1% of the benchmark.

What ‘defective’ means

The flaws vary: overly strict tests that fail correct solutions; under-specified prompts where it isn’t clear what’s even being asked; poor test coverage; and misleading prompts. In other words: a model can fail these tasks despite working correctly — or score without really solving the problem.

OpenAI’s conclusion: the community should have new benchmarks built directly by experienced developers, rather than automatically extracted from open-source pull requests.

My take

This matters to our readers for one reason: we all compare Claude, GPT and Grok using numbers like these. If a third of a widely cited benchmark is broken, then the pretty bar charts every new model is marketed with deserve caution too — including this week’s Grok 4.5 figures, which lean directly on SWE-Bench Pro.

I don’t read this as ‘benchmarks are worthless.’ More as a sign of maturity: the measuring instruments themselves are now being continuously audited, rather than treated as fixed truth. That’s healthy. Just pause the next time you see an ‘80% on SWE-Bench Pro’ post and ask: on which version, and how clean was the set?

Sources: