4arXiv cs.LG (Machine Learning)·
Benchmark shows Mondrian conformal prediction restores minority-class coverage in high-stakes imbalanced classification
A new arXiv preprint benchmarks cost-sensitive conformal prediction methods across 15 real-world imbalanced tabular datasets, 7 classifiers, and 3 calibration techniques (3,150 total runs), finding that standard marginal conformal prediction drops minority-class coverage to as low as 0.5%. Class-conditional (Mondrian) conformal prediction recovers valid minority coverage with an average 61.7 percentage-point improvement, and combining it with cost-controlled abstention reduces expected decision cost under realistic human review budgets. The paper also derives dataset-specific break-even thresholds for when deferring to human experts becomes cost-effective, offering practical deployment guidance for credit scoring, fraud detection, healthcare, and industrial safety applications.
Related guides (3)
Related events (8)
Theoretical framework for when synthetic data augmentation improves imbalanced classification metrics
A new arXiv preprint develops a theoretical framework characterizing when synthetic minority-class augmentation improves score-based metrics (AUROC, AUPRC, balanced accuracy, F1) under class imbalance. The authors show that under well-specified score models, augmentation provides no fundamental population-level improvement and may introduce bias, while under model misspecification it can correct ranking errors by shifting effective class balance. Minimax lower bounds confirm the raw estimator is already optimal in the well-specified regime, and simulation studies corroborate the theory.
Hindcast: A benchmark framework that closes data-leakage loopholes in LLM forecasting evaluation
A new arXiv preprint introduces Hindcast, an evaluation framework for LLM forecasters that addresses two systematic data-leakage channels in standard backtesting: post-event retrieval and training-data contamination from newer models. The system replays resolved Polymarket prediction markets against a frozen Reddit snapshot, grading models only on information available before a chosen cutoff date and comparing against contemporaneous market prices as a human-forecast baseline. Key finding: retrieval helps forecasting only when pre-event Reddit discussion existed; where only speculation was available, retrieval hurts performance. The framework is designed to remain valid as new models and markets emerge without going stale.
Risk-Aware Hybrid Selective Classification for HIV Suspicion Identification in Spanish Clinical Notes
This paper proposes a hybrid selective classification framework for clinical NLP that explicitly handles both aleatoric and epistemic uncertainty to avoid overconfident predictions in medical triage settings. The system combines Mondrian conformal prediction with a Multi-Centroid Mahalanobis Distance veto, evaluated on HIV suspicion identification in Spanish clinical notes. The authors demonstrate that standard uncertainty metrics and baseline classifiers suffer coverage collapse under strict reliability constraints, while their dual-verification approach isolates a trustworthy operational domain. The work critiques inflated benchmark metrics that arise from forcing deterministic classification on inherently ambiguous clinical instances.
Optimal deterministic multicalibration achieved, resolving open problem on randomization necessity
A new arXiv preprint resolves an open problem in multicalibration theory by constructing a minimax-optimal multicalibration algorithm that outputs a deterministic predictor, achieving the same O(ε⁻³) sample complexity previously only attainable by randomized predictors. The result extends to outcome indistinguishability, deterministic omnipredictors, and panpredictors with optimal sample complexity, resolving multiple open problems from recent works. Multicalibration is a fairness and reliability property requiring calibration to hold across reweighted subgroups, making this relevant to trustworthy ML research.
OptimismBench reveals systematic optimism bias in LLM probability judgments, linked to post-training alignment
Researchers introduce OptimismBench, a benchmark using inverted scenario pairs to detect directional forecasting bias in LLMs without requiring ground-truth probabilities. Across 16 models from 8 providers, 14 exhibit optimistic bias; pessimism appears only in Anthropic's frontier tier. Post-training (alignment/RLHF) is identified as the mechanism setting bias direction, with opposite shifts across model families, and the pattern is robust to prompt, temperature, and self-debiasing interventions. The finding has direct implications for downstream decision pipelines that rely on LLM probability estimates.
Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
This paper benchmarks six ensemble strategies across six tabular foundation models (TFMs) on 153 OpenML classification tasks, finding that ensembling provides minimal gains over the best single TFM. The best ensemble strategy (two-level cascade stacking) achieves only +0.18% accuracy improvement at 253× the compute cost. A key finding is that logistic-regression meta-learner stacking improves accuracy while severely degrading calibration (log-loss), because sharpening class boundaries destroys probability estimates. The authors recommend greedy ensemble selection as the practical default.
PosConf: Position-aware confidence calibration across SFT, RL, and OPD post-training stages
A new arXiv paper introduces a three-stage calibration framework analyzing how supervised fine-tuning (SFT), reinforcement learning (RL), and on-policy distillation (OPD) shape model confidence before, during, and after chain-of-thought reasoning. The authors find that each post-training method produces distinct calibration profiles at different reasoning stages, and that RL confidence becomes informative only after a path-commitment phase while OPD confidence degrades later. They propose PosConf, a position-aware confidence strategy that selectively uses confidence from reliable relative-position intervals, improving RL answer aggregation by 6.1 points over majority voting and OPD early stopping by up to 4.3 points.
RRBench: evaluating open-weight LLMs on longitudinal data preparation tasks in governance-restricted settings
Researchers from UCL introduce RRBench, an open-source benchmark and evaluation framework for assessing open-weight LLMs on data preparation tasks in longitudinal population studies, where cloud-based models are prohibited by data governance rules. The benchmark uses ground-truth cleaning scripts from six sweeps of a British cohort study, covering tasks like category harmonization and multi-wave merging, with automated evaluation of LLM-generated R code. Current 31–35B parameter open-weight models achieve up to 87.9% average task completion on consumer-grade hardware, suggesting a viable path for AI-assisted data preparation in privacy-sensitive research contexts.