5arXiv cs.CL (Computation and Language)·
LLM-powered automated extraction of 3.2M data points from 76,000 energy system studies
Researchers demonstrate automated extraction of quantitative techno-economic data from 76,000 energy system studies published since 2010, producing a FAIR database of 3.2 million structured data points and 20 million metadata entries. The system enables meta-analysis of the energy systems literature at scale, revealing where academic modeling assumptions diverge from empirical data. The work is a large-scale applied NLP/information-extraction deployment in a high-stakes scientific domain, with the resulting database made publicly accessible via an interactive dashboard.
Related guides (1)
Related events (8)
Two-stage LLM system extracts fine-grained events from 292K SEC 8-K filings with grounded citations
Researchers present a two-stage LLM pipeline that classifies SEC Form 8-K filings against a 119-type event taxonomy, anchoring every label to a verbatim quote via fuzzy n-gram validation and re-scoring each citation for quality. Applied to 292,984 filings from 2022–2026, the system produces 601,088 grounded event tags with precision rising from 12% to 96% as quality scores increase. The authors release the tagged dataset and validate economic signal via an event study on abnormal returns, confirming the taxonomy captures economically distinct events that the SEC's coarse item codes conflate.
Stanford EDGAR Filings Dataset: 152B-token open corpus of SEC filings for LLM pretraining
Stanford researchers introduce the Stanford EDGAR Filings Dataset (SEFD), an open reconstruction of SEC filings into layout-faithful MultiMarkdown, releasing a 152B-token initial snapshot with a larger 550B-token archive described. The dataset targets the growing scarcity of high-quality long-context pretraining data, with less than 0.1% overlap with Common Crawl-derived corpora. Two derived benchmarks are also introduced: EDGAR-Forecast for filing-grounded numerical forecasting and EDGAR-OCR for complex financial table transcription. The work addresses a real gap in open long-context training data outside narrow domains like code.
MetaSyn benchmark reveals critical screening bottleneck in LLM-based meta-analysis pipelines
Researchers introduce MetaSyn, a dataset of 442 expert-curated meta-analyses from Nature Portfolio journals, paired with a 140k-article PubMed retrieval corpus, PI/ECO criteria, verified positives, and hard negatives. Benchmarking twelve pipeline configurations — nine RAG variants and a protocol-driven agent — shows that despite 90.9% retrieval recall at K=200, no system recovers more than 52.7% of ground-truth included studies. The core failure is LLMs' inability to reliably distinguish eligible studies from topically similar but criteria-failing distractors. The paper argues that end-to-end scores obscure where pipelines break down and proposes stage-attributed metrics.
DataGovBench: Benchmarking LLMs on Real-World Governmental Data Analysis Tasks
Researchers introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs on practical data analysis tasks beyond simple fact retrieval from small tables. The benchmark covers two tasks: Table QA (complex decomposable questions with textual or visualization outputs) and Table Insight (exploratory data analysis for expert-level findings). Experiments with state-of-the-art LLMs, with and without agentic frameworks, reveal significant performance gaps, suggesting current systems fall well short of real-world data analytics demands.
Hyper-Extract: LLM-powered extraction of graphs, hypergraphs, and spatio-temporal structures from text
Hyper-Extract is a Python library that uses LLMs to transform unstructured text into structured knowledge representations including graphs, hypergraphs, and spatio-temporal extractions via a single command interface. The project is trending on GitHub with 1,723 stars and 124 new stars today. It targets a practical gap in the LLM tooling ecosystem for structured knowledge extraction beyond simple key-value or flat-schema outputs.
DECODEM: Benchmark for LLM extraction of corporate governance variables from legal documents
DECODEM introduces benchmark datasets pairing corporate charters and bylaws with human annotations for evaluating automated extraction of governance variables using LLMs. The paper evaluates multiple extraction pipelines varying in prompt design, task decomposition, and document handling, finding frontier models achieve high accuracy on most provisions. A key finding is that elaborate prompting and cascading pipelines do not consistently improve frontier model performance but substantially narrow the gap with smaller efficiency-oriented models, suggesting pipeline design can partially substitute for model capability in structured legal document extraction.
Structured LLM extraction of financial news outperforms sentiment-only approaches for stock prediction
Researchers propose a framework using LLaMA-3.1-70B to extract six semantic dimensions from financial news (event type, impact scope, temporal horizon, semantic confidence, etc.) beyond simple sentiment polarity. Experiments on 41,618 news-stock pairs from the FNSPID dataset show that combining LLM-extracted structured features with FinBERT sentiment achieves F1=0.600, significantly outperforming either alone, with a 53.5% systematic disagreement rate indicating the two signal sources are largely orthogonal. The work argues that compressing financial news to a single sentiment score incurs substantial information loss and that multi-dimensional NLP extraction is systematically exploitable for prediction tasks.
LLMs automate reproducibility assessments in social and behavioral sciences, outperforming human reanalysts
A preprint from arXiv demonstrates that an LLM pipeline can automate reproducibility assessments of published social and behavioral science studies, recovering original effect sizes in 41% of cases (vs. 34% for human reanalysts) and reaching the same qualitative conclusion in 96% of cases (vs. 74% for humans). The study evaluated 76 published studies with predefined claims. The results suggest LLMs could serve as a scalable tool for systematic auditing of empirical research, addressing the resource-intensive nature of traditional reproducibility efforts.