4arXiv cs.CL (Computation and Language)·
CreditCardQA: Financial literacy benchmark for numerical reasoning from real credit card agreements
Researchers introduce CreditCardQA, a 1,800-question benchmark derived from real credit card agreements to evaluate LLM numerical and financial reasoning. Evaluation across multiple models under Chain-of-Thought and Program-of-Thought prompting shows PoT consistently improves performance and narrows open/closed-source gaps. Error analysis reveals failures stem primarily from misapplied financial rules and misunderstood contractual terms rather than arithmetic errors, with edge cases disproportionately affecting financially vulnerable users.
Related guides (2)
Related events (8)
Benchmarking study finds LLMs fail at counterintuitive probability problems despite strong standard performance
A new arXiv paper evaluates 8 state-of-the-art LLMs on discrete probability problems using two datasets: standard exercises (average accuracy 0.96) and counterintuitive exercises designed to trigger heuristic reasoning (average accuracy 0.59). The authors document token bias causing 20%+ performance drops when canonical problem formulations are disguised, and up to 34% degradation when misleading suggestions are embedded in prompts. The findings argue that current LLMs are not genuine probabilistic reasoners despite their success on advanced math benchmarks.
NuclearQAv2: A benchmark for evaluating LLM competence in nuclear engineering
Researchers introduce NuclearQAv2, a ~1,240 question benchmark for assessing LLM performance on nuclear engineering knowledge across boolean, numeric, and verbal question types. The benchmark is constructed via a hybrid pipeline combining expert-authored questions, existing datasets, and LLM-assisted generation from domain-specific corpora. Evaluation of multiple LLMs reveals strong performance on factual recall but significant gaps in quantitative reasoning and conceptual understanding, highlighting the need for multi-faceted domain-specific evaluation.
TriviaRoomQA: Multilingual benchmark reveals LLM gaps in everyday popular-culture knowledge
Researchers introduce TriviaRoomQA, a multilingual benchmark with 3,300 parallel multiple-choice questions across six European languages and 5,340 French-only questions, covering 288 topics drawn from quiz and trivia contexts. Evaluating 30 open-weight LLMs (7B–70B parameters) from European, Asian, and North American providers, the study finds models perform well on history, geography, and math but substantially worse on popular-culture topics like celebrities, music, and movies. Performance also varies across languages for identical underlying questions, indicating factual knowledge access is language-dependent. The work highlights a knowledge gap not captured by existing saturated academic benchmarks.
CM-LRS: A capital markets reliability benchmark for LLM workflow outputs
Researchers introduce CM-LRS (Capital Markets LLM Reliability Score), a seven-dimension evaluation framework assessing LLM outputs at the workflow level rather than the question-answer layer, targeting regulated capital-markets use cases such as DCM/ECM term extraction, M&A comparables, and issuer profiling. The benchmark is demonstrated on five workflows using public SEC EDGAR and UK takeover filings, scoring four models across four LLM judges. Key findings: frontier closed-source models cluster tightly (Sonnet 4.6 = 4.31, Opus 4.7 = 4.30, GPT-5.5 = 4.09) while Llama 3.3 70B lags at 3.15, with the gap concentrated in retrieval and synthesis tasks rather than extraction. The work advances domain-specific evaluation methodology for high-stakes financial workflows where regulatory defensibility matters.
New Polish medical exam benchmark reveals MCQA overestimates LLM clinical competence
Researchers introduce an expanded Polish medical exam benchmark with over 15,000 new questions, two new domains, and four structural modifications designed to reduce multiple-choice artifacts and better test reasoning. Evaluating 21 LLMs under the harder setup, the best-performing model (Qwen3.5-122B) drops 28-31 percentage points compared to standard MCQA scores. The findings suggest standard MCQA benchmarks do not reliably reflect true medical competence, even when data contamination is low. The benchmark is publicly released to support further research.
MedQADE benchmark reveals LLM evaluators match physician agreement scores but lack clinical caution and show lineage bias
Researchers introduce MedQADE, a standardized open-response clinical benchmark for German comprising 3,800 items annotated by ten physicians and nine LLM evaluators. The top LLM evaluator (Gemini 3 Flash) reached statistical alignment near the physician inter-rater ceiling (κ=0.694 vs. κ=0.709), but automated evaluators showed near-zero clinical metacognition: unlike physicians, they never abstained regardless of item difficulty. The study also documents systematic lineage-dependent scoring bias, where models preferentially rate architectural siblings more favorably, independent of language.
AdvancedMathBench: New benchmark suite for advanced mathematical proof generation and verification
Researchers introduce AdvancedMathBench, a benchmark suite targeting advanced mathematical reasoning beyond olympiad-level problems, comprising ProverBench (296 undergraduate and doctoral qualifying-exam problems) and VerifierBench (888 model-generated proof trajectories). The suite includes an automatic verification pipeline trained on expert annotations to assess proof correctness and error types. Frontier models struggle significantly: the best performer (GPT-5.5-xhigh) achieves only 75.8% and 66.1% on undergraduate and qualifying-exam splits respectively, while proof verification tops out at 65.1 Balanced F1, with models particularly weak at detecting proof errors.
Qwen2.5-Math Process Reward Model for Mathematical Reasoning Supervision
Alibaba's Qwen team introduces a process reward model (PRM) aimed at improving the reliability of mathematical reasoning in LLMs by supervising intermediate reasoning steps rather than only final answers. The work addresses the problem of models producing plausible but flawed intermediate derivations even when reaching correct conclusions. The release includes model weights on HuggingFace and ModelScope alongside a GitHub repository.