HardEval: An Execution-Verified Benchmark for AI-Driven Security Hardening and Compliance Remediation
Provides rigorous evaluation methodology for AI security applications relevant to ML researchers and security professionals
AI Summary
Researchers introduce HardEval, an execution-verified for evaluating language models on security hardening and compliance remediation tasks.
Excerpt
Implementing Security Technical Implementation Guide (STIG) controls remains a demanding operating-system administration task. Each natural-language security requirement must be translated into a precise and persistent configuration change. These changes span Secure Shell (SSH) configuration, authentication policy, audit rules, and kernel parameters. Existing benchmarks do not measure whether a model can perform this task on a live system. Code benchmarks evaluate isolated programs without persi
