AMALIA

AMALIA-9B study finds LLM annotation agreement does not imply construct validity for moral foundation coding

A new arXiv paper tests whether Portugal's publicly funded AMALIA-9B language model can validly annotate moral foundation constructs (specifically authority) in European Portuguese text, not just agree with human coders. Using a 'recovery gap' methodology — comparing holistic prompting against decomposed codebook-clause prompting — the authors find AMALIA recovers only about half its holistic performance under decomposition, suggesting reliance on surface correlates rather than genuine construct understanding. A larger open multilingual model closes the same gap on the same corpus, pointing to model-level rather than corpus-level failure. The paper argues that sovereign/national LLM benchmark batteries should test the evidential route to agreement, not just agreement rates.