3GitHub Trending (AI/LLM filtered)·
FlashInfer: Kernel Library for LLM Serving trending on GitHub
FlashInfer is a kernel library targeting LLM serving workloads, currently at 6,018 GitHub stars with modest daily growth. It provides low-level GPU kernel implementations optimized for inference use cases. The project is gaining steady community traction in the inference infrastructure space.
Related guides (2)
Related events (8)
mlx-lm: LLM inference library for Apple MLX framework trending on GitHub
mlx-lm is an open-source Python library for running LLMs using Apple's MLX framework, designed for Apple Silicon hardware. The repository has accumulated 5,817 stars with 43 new stars today, indicating steady community interest. It represents a key piece of the Apple-native ML inference ecosystem.
vLLM: High-Throughput LLM Inference and Serving Engine Trending on GitHub
vLLM is an open-source Python library providing high-throughput and memory-efficient inference and serving for large language models. The project has accumulated over 80,500 GitHub stars with 98 new stars today, indicating continued strong community interest. It is a widely adopted inference backend in the AI/ML ecosystem, supporting PagedAttention and various optimization techniques for LLM deployment.
LMCache: KV cache layer for LLM inference acceleration
LMCache is an open-source Python library providing a KV cache layer designed to accelerate LLM inference. The project has accumulated 8,613 GitHub stars with modest daily growth (+17). It targets inference efficiency by offloading or sharing KV cache state across requests.
omlx: LLM inference server with continuous batching and SSD caching for Apple Silicon
omlx is an open-source Python project providing an LLM inference server optimized for Apple Silicon, featuring continuous batching and SSD caching managed via a macOS menu bar interface. The project has accumulated nearly 16,000 GitHub stars with strong daily momentum. It targets local inference on Apple hardware, a growing niche as consumer-grade silicon becomes increasingly capable for running open-weights models.
ktransformers: heterogeneous LLM inference/fine-tune optimization framework gains traction on GitHub
ktransformers is an open-source Python framework for heterogeneous LLM inference and fine-tuning optimizations, developed by kvcache-ai. The project has accumulated 18,236 GitHub stars with 328 added in a single day, indicating significant community interest. It targets KV-cache and mixed-hardware inference optimization, a key area for reducing LLM serving costs.
Langfuse: Open Source LLM Engineering Platform Trending on GitHub
Langfuse is an open-source LLM engineering platform providing observability, metrics, evaluations, prompt management, and dataset tooling. It integrates with OpenTelemetry, LangChain, OpenAI SDK, and LiteLLM. The project has accumulated 28,075 GitHub stars with 89 new stars today, indicating sustained community traction. Backed by Y Combinator (W23), it represents a notable entry in the LLM ops/tooling ecosystem.
Make your llama generation time fly with AWS Inferentia2
This Hugging Face blog post covers deploying and optimizing Llama 2 inference on AWS Inferentia2 accelerators. It demonstrates integration between Hugging Face's Optimum Neuron library and AWS's custom silicon to achieve competitive inference throughput and latency. The post serves as a practical guide for enterprise teams looking to reduce inference costs by moving off GPU-based infrastructure.
Microsoft BitNet: official inference framework for 1-bit LLMs trending on GitHub
Microsoft's BitNet repository, the official inference framework for 1-bit large language models, is trending on GitHub with over 39,000 total stars. The project enables efficient inference for extremely quantized models. Continued community interest signals ongoing relevance of 1-bit quantization as an inference efficiency approach.