6arXiv cs.CL (Computation and Language)·
Stemma: Black-box LLM fingerprinting via induced decision regions achieves 0.967 AUC on provenance testing
Researchers introduce Stemma, a black-box LLM fingerprinting method that determines whether a suspect model belongs to the same lineage as a source model. The approach maps open-ended outputs into a finite decision space ('induced decision regions') to abstract away surface-form variation caused by fine-tuning or deployment, then measures inheritance of those regions as a provenance signal. Evaluated across 770 source-suspect pairs from 56 public checkpoints, Stemma achieves 0.967 AUC and 87.8% TPR at 1% FPR, substantially outperforming four baselines, with even stronger results on deployment-setting robustness tests. The method addresses a practical gap in model IP protection and lineage auditing as fine-tuned derivatives proliferate.
Related guides (3)
Related events (8)
ProvenanceGuard: Source-aware factuality verification for MCP-based LLM agents
Researchers introduce ProvenanceGuard, a verifier that checks factual claims in MCP-grounded LLM agent answers against their specific source provenance rather than pooled evidence. The system decomposes answers into atomic claims, routes each to its attributed source via MCP trace metadata, and applies NLI plus token-alignment checks to detect 'cross-source conflation' — where a claim is supported somewhere but attributed to the wrong source. Evaluated on 281 medical-domain MCP-agent traces, it achieves block F1 of 0.802 and source accuracy of 0.858 on held-out data, and detects all injected attribution swaps in 50 controlled clinical probes. The work establishes source attribution as an independent factuality axis distinct from standard grounding checks.
AdversaBench: Automated LLM red-teaming pipeline with multi-judge confirmation and cross-model transferability
AdversaBench is a new end-to-end red-teaming pipeline that mutates seed prompts using five structured operators and confirms failures via a three-judge panel with a meta-judge tiebreaker. Experiments on 45 seeds across reasoning, instruction-following, and tool-use categories produced confirmed failures for every seed. Key findings include sharp variation in operator effectiveness by category, misleading binary failure rates, judge agreement metrics distorted by label skew, and zero-shot transferability of adversarial prompts from Llama 3.1 8B to Llama 3.3 70B. Code and dataset are publicly released.
IdeaGene-Bench: Benchmark for Scientific Lineage Reasoning and Idea Generation in LLMs
Researchers introduce IG-Bench, a benchmark evaluating whether AI systems can trace and reason about the evolutionary inheritance structure of scientific ideas across papers. The benchmark contains 1,961 lineage traces, 1,085 Idea Genome objects, and 920 pairwise GenomeDiff records spanning 10 scientific domains, supporting both closed-form reasoning tasks and lineage-conditioned generation evaluation. Experiments on 14 LLM-based systems reveal a compositional bottleneck: the best system achieves only 27.3% exact accuracy on lineage reasoning, and adding structured lineage context reshuffles system rankings rather than uniformly improving performance.
LCF: Linguistically-grounded code-mixing fingerprints for LLM ownership verification
A new arXiv preprint introduces LCF (Linguistically-Constructed Fingerprints), a unified framework for embedding ownership signals into LLMs via code-mixed triggers that combine low-resource languages under semantic-density substitution and grammar-biased mixing rules. The companion injection method, LCFEdit, uses null-space projection from multilingual representations to embed fingerprints while preserving model utility. The approach addresses two known weaknesses of prior fingerprinting methods: accidental activation of natural-language triggers and perplexity-based filtering of garbled triggers. Evaluations report persistent ownership verification with negligible utility degradation.
STRACE framework improves LLM agent optimization via structural trajectory analysis and causal extraction
Researchers introduce STRACE (Structural TRajectory Analysis and Causal Extraction), a framework for constructing high signal-to-noise optimization contexts for long-horizon LLM agents. At the batch level, STRACE mines failure patterns to filter redundant traces; within each trace, it performs causal localization over a textual dependency graph to isolate root-cause steps. On the formal verification benchmark VeruSAGE-Bench, STRACE achieves a 1.4× success-rate improvement (42.5% to 58.5%) over human-expert-designed agents, outperforming standard context-filtering baselines.
CWE-Trace framework reveals LLM vulnerability detection is calibration without comprehension
Researchers introduce CWE-Trace, a benchmark of 834 manually curated Linux kernel samples across 74 CWEs with strict temporal splits to prevent data contamination, used to evaluate 8 vanilla LLMs and 15 LoRA fine-tuned variants on vulnerability detection. Key findings: data contamination provides no measurable advantage (84% of nominally contaminated samples carry no usable memorization signal), and backbone directional priors dominate fine-tuning — models exhibit stable systematic failure modes that resist correction. The best binary detection score reaches only 52.1% (barely above chance) and exact CWE classification Top-1 accuracy stays below 1.3%, indicating fine-tuning shifts output distributions without instilling genuine security reasoning. The work introduces two diagnostic metrics (Directional Failure Index and Hierarchical Distance and Direction) and concludes that detection capability and security understanding are decoupled in current LLMs.
ModSleuth: Agentic system audits invisible dependency graphs in modern LLM training pipelines
Researchers introduce ModSleuth, an agentic system that recursively reconstructs LLM dependency graphs from public artifacts, recovering 1,060 source-verified dependencies across four major LLM releases. The system formalizes direct and indirect dependencies and operation-centered relationships to handle fragmented, inconsistent documentation. Applied at scale, the resulting graphs expose multi-hop license obligations, train-evaluation coupling, and discrepancies between released and training-time artifacts — issues that are practically invisible to manual auditing.
SIFT and WSP: Claim-conditioned re-scoring to close the warrant gap in LLM fact-checking
A new arXiv preprint identifies a 'warrant gap' in LLM-based fact-checking systems: models frequently output Supports verdicts whose cited evidence does not actually entail the claim. The authors introduce SIFT, a claim-conditioned re-scoring method for extracted evidence spans, and WSP (Warranted Supports Proportion), an automatic NLI-based metric that checks whether cited warrants entail the claim. Evaluated on FEVER, SciFact, 5PILS, and DP across four open-source backbones, SIFT recovers up to 27.6 accuracy points lost by naive decomposition, while WSP calibrates against human gold evidence at AUC 0.92 and precision 0.98.