Top AI Product

Microsoft open-sourced ThinkingBox, a sandbox plus benchmark that tests whether AI agents can be trusted with real business workflows. The twist: it ignores what the agent says and only checks what actually changed in the backend database.

One success isn’t reliability

The benchmark ran 507 policy-conditioned tasks across five domains — retail, hospitality, auto insurance, neobank IT, consulting support — with 20 repeated runs per task, across 12 models. The strongest model hit 65.36% pass@1. But pass^20, succeeding all 20 times on the same task, dropped to 25.25%. Your agent nails a task in the demo, then flubs it three out of four times in production. Worse: 80% of failed runs ended cleanly, looking like success. Response quality tells you nothing about whether the job got done.

Plug in any model

ThinkingBox is a framework, not just a leaderboard. Isolated MCP-compatible tool sessions, full execution traces, a judge loop, and a CLI. It supports Azure OpenAI, Anthropic, and any OpenAI-compatible endpoint, so you can benchmark your own agent stack or generate RL training data from the traces. MIT-licensed, paper on arXiv.

Every vendor claims their agent “works.” Microsoft built the machine that checks 20 times.

You Might Also Like

• Google Microsoft Salesforce Snowflake Servicenow Back one Agent Standard Aimed Straight at Anthropics mcp
• Openfang Just Dropped and its Already the Hottest Agent os on Github
• Agent Action Protocol aap the Missing Layer Above mcp That Actually Makes Agents Production Ready
• Insforge Hits 1 on Product Hunt and 3600 Github Stars is This What Agent Native Backends Look Like
• Openviking Treats ai Agent Memory Like a File System and 9k Github Stars say its Working