There’s been a lot of chest-thumping lately about AI coding agents solving real-world GitHub issues. SWE-bench scores keep climbing, and every new model launch comes with claims about “state-of-the-art” issue resolution rates. But here’s the thing — fixing a single bug in isolation is very different from maintaining a codebase over months. [SWE-CI](https://arxiv.org/abs/2603.03823) is a new benchmark that finally puts this distinction front and center, and the results are humbling.
The idea is simple but clever. Instead of handing an AI agent one issue and asking it to produce a patch, SWE-CI drops agents into a continuous integration loop. Each task spans real development history — on average 233 days and 71 consecutive commits across 68 actual open-source repositories. The agent has to iteratively analyze requirements, modify code, and pass tests, just like a real developer would during ongoing maintenance. The benchmark includes 100 carefully constructed samples, and the team introduces a metric called EvoScore that tracks how well an agent’s early decisions hold up as the codebase evolves. Make a quick hack now, and you pay for it ten commits later.
The findings? Even the best models tested — 18 in total — struggled badly. Around 75% of them broke something along the way. The top performers still caused regressions in roughly half their runs. It’s a stark reminder that current AI coding tools are great sprinters but terrible marathon runners.
SWE-CI [hit the Hacker News front page on March 8](https://news.ycombinator.com/) with 116 points and 40 comments, sparking a lively debate about whether we’ve been measuring AI coding ability all wrong. The paper is also trending on [Hugging Face Papers](https://huggingface.co/papers/2603.03823), and the full dataset is available on [Hugging Face Datasets](https://huggingface.co/datasets/skylenage/SWE-CI) under a CC BY 4.0 license.
What I find most interesting is the meta-question SWE-CI raises: are we building AI tools that accumulate technical debt faster than they resolve it? If an agent can fix your bug today but subtly degrades your architecture over time, is that actually useful? Worth thinking about before you hand your entire repo over to an autonomous coding agent.
Top AI Product 
Leave a comment