The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act

Benchmark gap paper: EU AI Act requires doctrinal legal reasoning evals that don't yet exist

A new arXiv preprint identifies a critical measurement gap in legal AI evaluation: existing benchmarks test paralegal and ancillary tasks rather than doctrinal legal reasoning, which is the interpretive core of legal work. The authors argue this gap is not merely methodological but legally significant, because the EU AI Act's 'appropriate accuracy' requirement for high-risk AI in the judicial domain cannot be operationalized without a doctrinal-reasoning benchmark. The paper proposes a benchmark framework aimed at filling this gap under EU AI Act compliance requirements.