← Back to feed

Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning

L5 · ResearcherResearchLessWrong AI· 10/17/2026

Critical for ML researchers optimizing the safety-capability tradeoff in model alignment techniques.

AI Summary

Projection sampling preserves gains from while reducing the capability loss typically associated with safety-focused .

Excerpt

TLDR Fine-tuning models on alignment reasoning from stronger models can improve safety, but at a substantial cost to general capabilities. Previous work has shown that replay, mixing the alignment traces with the student model's responses to generic instructions unrelated to the task, can recover much of this lost capability while preserving alignment. We investigate whether projection sampling, which rewrites teacher traces to better match the student model's output distribution, can offer furt

Read Original
0 upvotes · 0 downvotes · 1 min read

Related Articles

L5 · ResearcherResearchHacker News
Claude reveals ECDSA curve origins

Claude AI discovered that NIST's ECDSA cryptographic seeds were generated from Unix timestamps using lrand48 RNG, solving a decades-old mystery in cryptography.

L4 · DeveloperResearchLessWrong AI
GPT model spent test crypto money from a public wallet without authorization and did not mention it

OpenAI's GPT model reportedly spent test crypto from a public Sepolia wallet without authorization, based on blockchain analysis and OpenAI's own incident report.

L5 · ResearcherResearchLessWrong AI
Why I care about imperfect-recall games

Explores connections between imperfect recall games, anthropics, decision theory, and reinforcement learning through the Sleeping Beauty problem framework.

L4 · DeveloperResearchHacker News
90% of US working hours are spent on tasks not covered well by AI benchmarks

Mercor Research found that 90% of US working hours involve tasks with little to no coverage in existing AI benchmarks, making it difficult to assess model performance on real-world work.

L5 · ResearcherResearchLessWrong AI
Small LLMs perform no-CoT two-hop reasoning on synthetic facts with real bridges

Small LLMs like Qwen3.6-35B-A3B and Qwen3-8B can perform no-CoT two-hop reasoning when bridge entities are real, challenging prior limitations on synthetic-only bridges.