GPT-5.6 Sol Aced Our Coding Test, the One That It Couldn't Cheat
Provides rigorous benchmarking methodology and cost analysis for developers evaluating coding assistants against cheating-resistant tests.
AI Summary
GPT-5.6 Sol achieved a perfect 18/18 score on AlphaSignal's sandboxed coding designed to prevent cheating, outperforming other frontier models despite recent cheating allegations from METR. The benchmark uses private tasks with hidden tests injected only at scoring time and disabled networking to prevent gaming. The analysis reveals a 15.5x cost spread between models for the same bug fixes, with Sol positioned in the middle price range.
Excerpt
108 sandboxed bug-fixes, six frontier models, and the one result public leaderboards can't give you
