Large-scale Near-deduplication Behind BigCode
This Hugging Face blog post details the near-deduplication pipeline developed for the BigCode project, which processes large-scale source code datasets used to train code language models. The post covers the technical methodology for identifying and removing near-duplicate documents at scale, including hashing techniques and distributed processing approaches. Deduplication is a critical preprocessing step that affects training data quality and model generalization.
Related guides (3)
Related events (8)
Evaluating Large Language Models Trained on Code
OpenAI published research on evaluating large language models trained on code, introducing the Codex model and the HumanEval benchmark for assessing code generation capabilities. The work established foundational methodology for measuring functional correctness of code produced by LLMs using a pass@k metric. This paper became a landmark reference for code-focused LLM evaluation and influenced subsequent code generation research across the field.
BigCodeBench: The Next Generation of HumanEval
Hugging Face introduces BigCodeBench, a new code generation benchmark designed to succeed HumanEval by offering more challenging and diverse programming tasks. The benchmark aims to better evaluate LLMs on real-world coding scenarios involving complex function calls and library usage. A leaderboard accompanies the release to track model performance across the community.
Streaming Datasets: 100x More Efficient
Hugging Face published a blog post describing efficiency improvements to their datasets streaming functionality, claiming up to 100x gains. The post covers technical changes to how large datasets are accessed and loaded without full downloads. This is relevant to ML practitioners working with large-scale training data pipelines.
Scaling AI-based Data Processing with Hugging Face + Dask
Hugging Face published a blog post describing how to scale AI-based data processing pipelines by combining Hugging Face datasets and models with Dask, a parallel computing framework. The post covers patterns for distributed inference and large-scale dataset preprocessing. This is a practical integration guide targeting ML engineers who need to process data at scale beyond single-machine limits.
Scaling Robotics Datasets with Video Encoding
Hugging Face published a blog post on using video encoding techniques to scale robotics datasets. The post addresses the practical challenge of storing and transmitting large-scale robot learning data efficiently. Video compression is presented as a key infrastructure enabler for expanding robotics training corpora.
CodeGraph: Pre-indexed Local Code Knowledge Graph for AI Coding Agents
CodeGraph is an open-source TypeScript tool that builds a pre-indexed knowledge graph of a codebase to reduce token usage and tool calls for AI coding agents including Claude Code, Codex, Cursor, OpenCode, and Hermes Agent. It runs entirely locally, positioning itself as an efficiency layer between codebases and LLM-based coding assistants. The project gained significant traction with 3,688 stars in a single day, reaching 16,371 total stars.
Deep Learning over the Internet: Training Language Models Collaboratively
This Hugging Face blog post describes a framework for training large language models collaboratively across volunteer compute contributed over the internet. The approach addresses the challenge of enabling distributed participants with heterogeneous hardware to jointly train models without centralized infrastructure. It represents an early exploration of decentralized training as an alternative to large-scale private compute clusters.
StarCoder: A State-of-the-Art LLM for Code
Hugging Face and ServiceNow released StarCoder, a large language model for code trained on permissively licensed data from The Stack dataset. The model targets code generation, completion, and understanding tasks and is positioned as an open-weights alternative to proprietary code models. The release includes model weights, training details, and an associated technical report.