FDE-Gym: The Last Mile of AI Delivery
Provides rigorous evaluation framework for AI systems handling complex real-world deployment scenarios beyond well-specified tasks.
AI Summary
Researchers introduce FDE-Gym, a simulating real-world AI delivery tasks where agents must clarify ambiguous requirements and discover unstated constraints in customer environments.
Excerpt
Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated cu
