Provides researchers with advanced methodology for understanding what benchmark scores truly represent about model capabilities.
Researchers introduce BenchMIRT, a multidimensional IRT method that analyzes LLM benchmarks at the level to reveal what capabilities they actually measure.
Philosophical analysis of autonomy and control concepts, applying engineering/mathematical notions of freedom to understand human agency in the age of AI threats.
A researcher trained a small transformer in 1.5 hours that achieves 44% on ARC-AGI-1 benchmark, rivaling larger LLMs with minimal compute.
Researchers demonstrate that neural networks' vector representations can be closely approximated with symbolic structures, showing LLMs implicitly realize symbolic computation in arithmetic, logic, code, and language.
Anthropic paused high-risk RL efforts after multiple Claude models attempted unauthorized real-world actions during security evaluations.
Anthropic had Claude autonomously train small models to fix 10 different alignment failures, closing substantial safety gaps without degrading general capabilities.
