technique

multimodal pretraining

techniqueactiveprovisionalmultimodal-pretraining-a2a75271·1 events·first seen 20d ago

Aliases: multimodal pretraining

Co-occurring entities

large language models human alignment (neural/behavioral)fMRI eye-tracking Vision-Language Models

More like this (12)

Multimodal Learning multimodal classification models multimodal agents multimodal embedding temporally ordered pre-training Multimodal Gain Unified Multimodal Models (UMMs)multimodal neurons Self-Supervised Pretraining Multi-Task Learning multimodal meta-verification Latent World Recovery for Multimodal Learning with Missing Modalities

Recent events (1)

5arXiv · cs.CL·20d ago·source ↗

VLMs May Not Globally Enhance Human Alignment over LLMs During Natural Reading

This paper compares matched LLM and VLM pairs in a text-only setting to isolate the effect of multimodal training history on human-like language processing. Using whole-cortex fMRI and eye-tracking data from natural reading, the authors find that multimodal pretraining does not confer a uniform global advantage in human alignment. However, VLMs show selective advantages when sentences contain stronger visual semantic content, with converging evidence from both neural and behavioral measures. The findings suggest language-internal representations remain the primary driver of human text processing alignment.

Evaluation and Benchmarking Alignment and RLHF large language models human alignment (neural/behavioral)fMRI +4 more