In-context learning applied to Multiple Instance Learning via Perceiver-style pretraining on synthetic data
A new arXiv preprint proposes pretraining an in-context learner with a Perceiver-style architecture on synthetic bag-structured data to solve Multiple Instance Learning (MIL) tasks from a handful of labeled bags at inference time, requiring no gradient updates. The authors evaluate several synthetic data generators and find that a mixture-pretrained model captures complementary inductive biases, outperforming supervised baselines across twelve MIL benchmarks. The work addresses the low-label regime common in domains like computational pathology and satellite imagery.
Related guides (1)
Related events (8)
Multi-Task Bayesian In-Context Learning for Amortized Hierarchical Inference
A new arXiv preprint introduces a multi-task in-context learning framework for amortized hierarchical Bayesian predictive inference, representing prior information as a prefix of in-context datasets fed to a transformer. The model learns to adapt predictions across families of priors, addressing the brittleness of prior-data fitted models under distribution shift. On evaluations including out-of-meta-distribution priors and high-dimensional latent structures, the method matches oracle Bayesian predictors while being orders of magnitude faster, with a real-world spatiotemporal temperature prediction demonstration.
ContextRL: Context-aware reinforcement learning improves grounding in agentic and multimodal LLMs
Researchers introduce ContextRL, a reinforcement learning method that trains LLMs to select the context that supports a given query-answer pair from two highly similar candidates, rather than supervising only final answers. The approach constructs contrastive context pairs in two domains: coding agent trajectories (1k pairs) and multimodal image pairs (7k pairs). ContextRL achieves +2.2% average gains over standard GRPO on 5 long-horizon benchmarks and +1.8% across 12 visual QA benchmarks, with ablations showing the gains stem from the context-selection objective rather than the contrastive data alone.
Vision-OPD: On-Policy Self-Distillation for Fine-Grained Visual Understanding in MLLMs
Vision-OPD addresses a 'regional-to-global perception gap' in multimodal LLMs, where models answer fine-grained visual questions more accurately when given cropped evidence regions than full images. The method instantiates a crop-conditioned teacher and full-image-conditioned student from the same MLLM, minimizing token-level divergence along on-policy rollouts to transfer regional perception to the full-image policy. This self-distillation requires no external teacher models, ground-truth labels, reward verifiers, or inference-time tools. Benchmarks show competitive or superior performance against larger open-source, closed-source, and agentic 'Thinking-with-Images' models.
Latent World Recovery: multimodal learning framework for missing modalities in bioscience
A new arXiv preprint introduces Latent World Recovery (LWR), a framework for multimodal learning when some modalities are unavailable at training or inference time. LWR aligns modality-specific embeddings in a shared latent space and fuses only available modalities, avoiding explicit reconstruction of missing ones. The approach is evaluated on incomplete multi-omics benchmarks for cancer phenotype classification and survival prediction, demonstrating robustness under partial observation.
Cosmopedia: Creating Large-Scale Synthetic Data for Pre-training LLMs
Hugging Face introduces Cosmopedia, a large-scale synthetic dataset designed for pre-training large language models. The blog post details the methodology for generating diverse, high-quality synthetic text at scale using existing LLMs as data generators. The work addresses the growing challenge of data scarcity and quality in LLM pre-training pipelines.
Synthetic data bootstrapping and LoRA fine-tuning for Q'eqchi' Mayan NMT without web scraping
Researchers introduce a data synthesis methodology for low-resource neural machine translation of Q'eqchi' Mayan, converting community-sourced dictionaries into a synthetic parallel corpus to avoid scraping target-language data. Using LoRA adapters on mT5-base, the approach achieves BLEU 42.02 on in-domain evaluation but only 0.59 against organic text, revealing a structural-semantic gap. An ablation with multi-task learning produced negative transfer, suggesting LoRA capacity limits conflict with auxiliary objectives. The study concludes synthetic bootstrapping is effective for structural priming but requires authentic data for semantic refinement via curriculum learning.
Cross-lingual in-context learning source language selection challenges fine-tuning assumptions
A new arXiv paper conducts a broad empirical study of cross-lingual transfer in few-shot in-context learning (ICL), spanning seven tasks, six models, and a typologically diverse set of languages. The study finds that conventional heuristics from supervised fine-tuning — such as relying on linguistic similarity or data availability — do not consistently transfer to the ICL regime. The authors also analyze language confusion as a key obstacle in generative cross-lingual ICL and propose alternative heuristics for source language selection.
Active Query Synthesis for Preference Learning via Mutual Information Maximization
This paper introduces Info-Synth, an active query synthesis framework for preference learning that generates optimal pairwise queries by maximizing a mutual information objective in continuous space, bypassing the computational cost of pool-based evaluation. A confidence-aware response model is proposed to handle ambiguous comparisons between nearly identical or highly dissimilar items. Two finite-pool extensions (Pair M-dist and Pair Opt-dist) are also introduced. The framework is validated on synthetic preference tasks, text summarization datasets, and robotic controller tuning.