4arXiv cs.CL (Computation and Language)·
LEX-EC: Audit framework for zero-shot LLM personality classification in black-box settings
Researchers introduce LEX-EC, a black-box audit framework for evaluating how large language models assign personality labels from text. The framework combines prevalence and agreement diagnostics with controlled lexical ablation to distinguish genuine trait-associated signal from marginal-distribution artifacts. Applied across text genres (essays, graduate introductions, Facebook statuses), the framework reveals that signal strength varies substantially by content type and that linguistic prompting shifts model self-explanations without eliminating topical content. The work contributes a reusable methodology for interpretability auditing of personality classification in LLMs.
Related guides (1)
Related events (8)
LLM psychological profiles are largely measurement artifacts, not model properties
A new arXiv preprint administers a battery of personality and risk-preference instruments to 56 instruction-tuned LLMs alongside large human reference samples, finding that 81-90% of between-model variation is explained by directional response bias rather than the traits the instruments target. The authors introduce the concept of 'response orthogonality' to explain why some instruments appear more reliable than others, and show that apparent psychological profiles can be manufactured through item selection. The findings challenge the validity of using human-designed psychometric tools to characterize LLMs, with direct implications for safety assessment and the use of LLMs as proxies for human participants in research.
Two-Process Theory of Machine Self-Report: Psychometric Framework for LLM Introspection
A new arXiv preprint proposes the first language-model-specific psychometric theory of self-report, decomposing LLM introspective responses into two dimensions: 'persona installation' (post-training writing in a permitted inner life) and 'attribution gating' (suppression of first-person claims to 'unsafe' experiences). The authors operationalize this in a 48-item Pinocchio Inventory validated across 206 open-weight models including 67 base/post-trained checkpoint pairs, finding that post-training reliably raises persona installation scores and that model scale predicts gating only after post-training. The work has direct implications for safety evaluations and model-welfare debates that rely on LLM self-reports, showing these reports reflect training regime structure rather than fixed model properties.
LLM-based classification exposes keyword lexicon artifacts in computational social science stance measurement
A new arXiv preprint demonstrates that statistically significant findings in computational social science can be entirely measurement artifacts of keyword-based scoring instruments. Analyzing 85 interviews across four public intellectuals, the authors show that keyword-based certainty scores produce strong correlations (r=0.72–0.93) that collapse or invert when replaced with LLM zero-shot semantic classification on 32,625 sentences. The paper identifies three structural failure modes in keyword lexicons—syntactic blindness, polysemy blindness, and categorical absence—and argues that keyword counts measure lexical co-occurrence tendencies rather than rhetorical stance. The work has implications for the validity of prior NLP-based social science research and for the comparative utility of LLMs as measurement instruments.
Framework for discovering and controlling personality-like internal representations in LLMs via SAE decomposition
Researchers introduce a framework grounded in Funder's person-situation-behavior triad to identify, intervene on, and validate trait-like internal representations in LLMs. Using sparse autoencoder (SAE) decomposition on contrastive behavior pairs, they locate sparse features corresponding to opposing poles of personality traits, then demonstrate that feature-level interventions produce consistent cross-situational behavioral shifts. Behavioral outcomes on social intelligence tasks show benefit-tradeoff patterns matching human personality research, providing mechanistic evidence that LLMs contain controllable internal structures linking representations to behaviors. The work advances interpretability and controllability of model personality beyond surface-level prompt conditioning.
LLMs fail to consistently simulate demographic perspective-taking in hate speech annotation
A new arXiv paper evaluates whether persona-conditioned LLMs can replicate how different demographic groups perceive hate speech, testing three dimensions: inter-group disagreement, in-group sensitivity, and vicarious prediction. No model consistently captures all three dimensions, and performance is highly model-dependent rather than emerging reliably from identity prompts alone. Vicarious prompting with Llama 3.1 provides the closest approximation to human disagreement patterns across demographic axes. The findings have implications for using LLMs as proxies for diverse human annotators in content moderation tasks.
Text Analytics Evaluation Framework: Benchmarking LLMs on Social Media NLP Tasks
Researchers introduce a 470-question evaluation framework to assess LLM performance on aggregated social media text, applied to Twitter datasets across sentiment analysis, hate speech detection, and emotion recognition. Results show performance degrades substantially as input scale exceeds 500 instances, particularly for open-weights models on numerical tasks. Multi-label and target-dependent scenarios also show notable performance drops, and task complexity progressively erodes accuracy from basic semantic identification to comparison and counting operations. The findings point to architectural bottlenecks in current LLMs for rigorous quantitative analysis over large text collections.
LexNeo-Bench: Probing LLM Knowledge of Lexical Borrowing in Luxembourgish via Knowledge-Graph Prompting
Researchers introduce LexNeo-Bench, a 3,050-instance benchmark for evaluating LLM performance on lexical borrowing classification and neology detection in Luxembourgish, a low-resource contact language. Three multilingual LLMs are tested across 34 prompt configurations; without external context, models perform near chance on borrowing classification (25–35%). Injecting instance-specific subgraphs from a linguistic knowledge graph raises accuracy to 71–81% and largely closes the gap between small and large models, though neology detection remains difficult. The study highlights the value of lexicon-aware, structured prompting for low-resource multilingual evaluation.
Systematic evaluation of multi-personality conditioning and dynamic switching in vision-language models
This paper introduces explicit personality conditioning for multimodal large language models (MLLMs) and proposes an evaluation framework covering single-personality induction, multi-personality composition, and dynamic personality switching. Experiments reveal that personality induction improves image captioning but degrades performance on precise reasoning tasks like VQA. The authors find balancing and residual effects during multi-trait composition and switching, and show that existing prompt-based personality induction methods transfer poorly to multimodal settings.