4arXiv cs.CL (Computation and Language)·
Adversarial robustness evaluation of Arabic language models across attack granularities
A new arXiv preprint evaluates five Arabic language models (including MARBERT and AraBERT) against adversarial attacks at character, word, and sentence levels, finding that diacritic insertion can reduce model accuracy by up to 92% and paraphrasing attacks degrade performance by an average of 76%. The study also tests adversarial training as a defense, finding MARBERT most robust and AraBERT showing the greatest relative gains. Results highlight particular challenges for morphologically rich languages where character-level noise remains difficult to defend against.
Related guides (2)
Related events (8)
Multilingual study finds LLMs are not uniformly robust to non-canonical tokenizations, with up to 23.7% performance drops
A new arXiv paper investigates how language models behave when given alternative (non-canonical) tokenizations of the same input string across 27 languages and six downstream tasks. While prior work showed English models are largely invariant to such perturbations, the study finds this does not generalize: Llama-3.1-8B drops 23.7% on average, Qwen3-8B 11.4%, and Gemma-3-12B 9.9% in relative performance. Languages with higher token fragmentation are systematically more sensitive, and the authors show LoRA fine-tuning on multi-tokenization data—including English-only data—provides meaningful mitigation.
Study exposes generalization failures in Bangla hate speech detection models
A new arXiv paper evaluates six NLP architectures for Bangla hate speech detection, finding severe performance degradation when models trained on benchmark datasets are applied to real-world social media content. BanglaBERT drops from 91.4% F1 on benchmarks to 63.4% on implicit hate speech involving sarcasm and emojis. The study highlights a broader generalization crisis for low-resource language moderation systems and finds that emoji-aware preprocessing recovers up to 12% of lost performance.
Adversarial robustness and safety alignment in multilingual multimodal LLMs: cross-lingual vulnerability and 'safety-by-failure'
A systematic study evaluates adversarial robustness and safety alignment of multimodal LLMs across 12 languages, finding that adversarial images optimized in one language transfer to others (cross-lingual transferability). The paper introduces the concept of 'safety-by-failure': low-resource languages appear safer not due to genuine alignment but because models fail to comprehend harmful instructions in those languages. Models like Qwen3-VL that integrate multilingual capability throughout training (rather than only at instruction tuning) show genuine cross-lingual safety with active refusal. The findings challenge the assumption that low-resource language safety metrics reflect real alignment.
Testing Robustness Against Unforeseen Adversaries
OpenAI published a method to evaluate whether neural network classifiers can defend against adversarial attacks not encountered during training. The approach introduces a new metric called UAR (Unforeseen Attack Robustness) to quantify a model's resilience to unanticipated attacks. The work argues for measuring robustness across a broader, more diverse set of attack types rather than only those seen in training.
Arabic Leaderboards: Introducing Arabic Instruction Following, Updating AraGen, and More
Hugging Face introduces new Arabic-language evaluation infrastructure, including an Arabic Instruction Following benchmark and updates to the AraGen leaderboard. The post covers evaluation methodology for Arabic LLM capabilities, expanding the ecosystem of non-English benchmarks. This is part of a broader effort to track model performance on Arabic language tasks beyond standard English-centric evaluations.
Unified defense framework detects and remediates data poisoning in text summarization fine-tuning
A new arXiv preprint introduces a post-hoc defense framework for detecting and recovering from training-time data poisoning in LLMs fine-tuned for abstractive summarization. The framework uses influence-function analysis in white-box settings and behavioral perturbation auditing in black-box settings, achieving 85-92% detection precision across nine architectures and six benchmarks. Gradient-ascent unlearning restores up to 96% of original model behavior with less than 0.6% ROUGE degradation. The authors also introduce novel attacks targeting factual distortion and representational bias that evade conventional evaluation metrics.
Falcon-Arabic: A Breakthrough in Arabic Language Models
TII UAE has released Falcon-Arabic, a language model specifically designed for Arabic. The announcement highlights it as a significant advancement in Arabic NLP capabilities. As a tier-2 source with minimal body content, specific technical details about model size, training data, or benchmark performance are not available from this item.
Artificial Aphasias in Lesioned Language Models
Researchers apply an aphasia-inspired 'lesioning' technique to five 1B-scale language models by zeroing out model parameters and measuring resulting language impairments against a Text Aphasia Battery (TAB). Across 112,426 outputs, the full range of aphasia symptoms emerges but in distributions distinct from human aphasia profiles. The study finds systematic differences between attention components (query, key, value, output) and feed-forward components, as well as depth-dependent effects where early-layer lesions cause syntactic/semantic symptoms and late-middle layers yield phonological and fluency deficits. The qualitative divergence between LM and human aphasia patterns suggests aphasia syndromes are shaped by learning and processing details rather than being universal consequences of disrupted language processing.