FisherSketch

FisherSketch: Efficient Fisher Alignment for Training-Free Source Selection in LLM Fine-Tuning

A new arXiv preprint introduces FisherSketch, a method for estimating head Fisher alignment between LLM tasks using a streaming, memory-efficient sketch (16 KB task signature, 192 KB streaming state) without materializing the full Fisher matrix. The work targets training-free source corpus selection for LLM families with shared vocabularies, particularly in scientific string domains like SMILES, protein, and genomic sequences. The authors demonstrate that representation similarity metrics like CKA are non-identifiable for transfer in shared-output-head settings, and validate FisherSketch on Llama-3.1-8B with verbalizer-shift experiments.