GPT-5.6 Sol can crack open math problems, yet it scored a laughable 13.3% on ARC-AGI-3, a set of 2D puzzle games. On July 31 OpenAI published the autopsy, and the culprit wasn’t the model.
The harness was starving the model
The official ARC-AGI-3 runner throws away the model’s reasoning after every single move, and once context fills up it deletes the earliest actions. So the agent keeps waking up amnesiac, re-deriving the game from scratch every turn. Flip on two switches in the Responses API — retained reasoning (carry the chain of thought across turns) and compaction (summarize old context instead of dropping it) — and the score jumps to 38.3%. It also spent 6x fewer output tokens.
Why devs care
This isn’t a new model launch. It’s OpenAI admitting a benchmark measures your harness as much as your weights. Any developer building agents can reproduce it today: same two parameters, same API, no fine-tuning. The uncomfortable takeaway — good context engineering can beat several model upgrades stacked together.
You Might Also Like
- Gemini 3 Deep Think Scores 84 6 on arc agi 2 Googles Reasoning bet is Paying off
- Openai gpt Realtime 2 Translate Whisper Three Voice Models one api Several Startups Erased
- Openai gpt 5 6 sol Terra Luna a Three Tier Lineup Only 20 Orgs can Touch
- Openai gpt Realtime 2 1 gpt Realtime 2 1 Mini cut Voice Agent Latency by 25
- Openai gpt 5 6 Same Smarts Half the Price sol Hits 750 Tokens s on Cerebras
Top AI Product 
Leave a comment