QF3: Fast Flow RL with Filtered Q-Gradients
Presents novel RL algorithm with significant performance improvements for robotics applications, relevant for ML researchers working on robot learning.
AI Summary
QF3 introduces a new off-policy reinforcement learning algorithm that trains flow policies 10x faster than previous methods for robot locomotion and manipulation tasks.
Excerpt
Flow policies have become a standard policy class for learning robot behaviors from demonstrations, but reinforcement learning is still critical for improving pre-trained flow policies or learning them from scratch through interaction. We introduce QF3 (Fast Flow RL with Filtered Q-Gradients), an online off-policy RL algorithm that trains a flow policy with flow matching plus the critic's action gradient, backpropagated through a one-step prediction of the flow's output. To keep updates where th