Top AI Product

Everyone ships a coding model with a shiny SWE-bench number attached. OpenAI just published a technical post arguing most of those numbers are noise. It hit 216 points on Hacker News, and the reason is obvious: it’s OpenAI throwing cold water on the exact scoreboard the whole industry uses to declare victory.

What it actually says

This isn’t a product. It’s a blog post — “Separating signal from noise in coding evaluations” — that dissects why coding benchmarks lie. The uncomfortable finding: run the same model on the same benchmark multiple times and scores can swing by ten-plus points. That means a lot of “new SOTA” announcements sit inside the measurement error. Nobody moved the needle; they got a lucky roll.

OpenAI traces the noise to a few sources. Contamination, because benchmark tasks come from public GitHub repos the models already trained on. Randomness in sampling. And a design flaw: unit tests inside a pull request were written to validate one specific human fix, not to define an implementation-agnostic “solved.” A correct-but-different solution fails. OpenAI even retracted its own earlier nod to SWE-Bench Pro after finding the same rot.

Why it matters

The whole “signal” here is trust. When bad evals feed capability and safety decisions, you’re steering with a broken gauge. Coming from the company that benefits most from big benchmark headlines, that’s a pointed admission — and a challenge to read the next “beats every open model” press release with a skeptic’s eye.

You Might Also Like

• Ornith 1 0 Deepreinforce Self Scaffolding Coding Models Open Weights That Write Their own rl Scaffold 397b Hits 82 4 on swe Bench
• Openai Symphony Finally a Framework That Lets you Stop Babysitting Your Coding Agents
• Swe ci Exposes What ai Coding Agents Still Cant do
• Cursor Composer 2 Takes on Anthropic and Openai With a 0 50 m Token Coding Model and the Benchmarks Back it up
• Stripe Coinbase and Ramp Built Internal Coding Agents Langchain Open swe Gives you the Same Architecture for Free