technique

format-aware task prototypes

techniqueactiveprovisionalformat-aware-task-prototypes-5c708469·1 events·first seen 15d ago

Aliases: format-aware task prototypes

Co-occurring entities

Multimodal Large Language Models ProtoAda LoRA Mixture of Experts Multimodal Continual Instruction Tuning

More like this (12)

task-agnostic generation tool-calling agents error-aware specialization objective topic modelling agent-task efficiency language-aware adapter heads prompt-optimizer PGT (Procedurally Generated Tasks)tool-augmented language agents Knowledge-Augmented Tool Execution Preference-Aware Rubric Learning Doc-to-Atom

Recent events (1)

5arXiv · cs.LG·15d ago·source ↗

ProtoAda: Prototype-Guided Adaptive Adapter Expansion for Multimodal Continual Instruction Tuning

ProtoAda is a new framework for Multimodal Continual Instruction Tuning (MCIT) that addresses a key failure mode in sparse Mixture-of-LoRA-Experts architectures: image-text similarity routing is format-blind and incorrectly merges tasks with similar semantics but different output structures (e.g., coordinate prediction vs. VQA). The method introduces format-aware task prototypes to guide both routing and adapter expansion, then consolidates compatible updates geometrically to reuse and refine existing parameters. Experiments across multiple benchmarks show improved performance, particularly on tasks whose answer formats are vulnerable to corruption by sequential fine-tuning.

Agent and Tool Ecosystem Alignment and RLHF Multimodal Large Language Models ProtoAda LoRA +4 more