Run a ChatGPT-like Chatbot on a Single GPU with ROCm
Hugging Face published a guide demonstrating how to run a large language model chatbot on a single AMD GPU using ROCm, AMD's open-source GPU compute stack. The post covers setup, model loading, and inference on AMD hardware as an alternative to NVIDIA CUDA-based workflows. This is relevant to the growing interest in democratizing LLM inference beyond NVIDIA's ecosystem.
Related guides (3)
Related events (8)
AMD + Hugging Face: Large Language Models Out-of-the-Box Acceleration with AMD GPU
Hugging Face and AMD announced integration work enabling out-of-the-box LLM acceleration on AMD GPUs via the Optimum library. The collaboration targets ROCm-based AMD hardware, aiming to reduce friction for users running inference on non-NVIDIA GPU stacks. This represents a continued push to broaden the hardware ecosystem available to open-weights model users.
Easily Build and Share ROCm Kernels with Hugging Face
Hugging Face has published a guide and tooling for building and sharing custom ROCm kernels on its platform, targeting AMD GPU users in the ML ecosystem. The post covers the workflow for packaging, uploading, and reusing ROCm-based GPGPU kernels via the Hub. This lowers the barrier for AMD GPU kernel development and sharing, complementing the existing CUDA-centric kernel ecosystem. The initiative is relevant to inference optimization and the broader push to diversify GPU hardware support in AI workloads.
Fast Inference on Large Language Models: BLOOMZ on Habana Gaudi2 Accelerator
This Hugging Face blog post covers deploying BLOOMZ, a large multilingual language model, on Intel's Habana Gaudi2 accelerator for inference. It benchmarks throughput and latency performance on Gaudi2 as an alternative to GPU-based inference. The post is part of ongoing work to demonstrate non-NVIDIA hardware options for large model deployment.
Hugging Face on AMD Instinct MI300 GPU
Hugging Face announces support and optimization for AMD Instinct MI300 GPUs, expanding the ecosystem of hardware that can run Hugging Face models and tools. The post covers integration work enabling inference and training workloads on AMD's high-memory GPU accelerator. This represents a meaningful step in diversifying AI infrastructure beyond NVIDIA dominance.
A Chatbot on your Laptop: Phi-2 on Intel Meteor Lake
This post demonstrates running Microsoft's Phi-2 small language model locally on Intel Meteor Lake laptop hardware. It covers the inference pipeline, optimization techniques, and performance characteristics of deploying a 2.7B parameter model on consumer-grade NPU/CPU hardware. The piece highlights the growing feasibility of on-device LLM inference without cloud dependency.
Text-Generation Pipeline on Intel® Gaudi® 2 AI Accelerator
Hugging Face published a blog post detailing how to run text-generation pipelines on Intel's Gaudi 2 AI accelerator. The post covers integration between Hugging Face's text-generation tooling and Intel's Gaudi 2 hardware, positioning it as an alternative inference accelerator to NVIDIA GPUs. This is relevant to the growing ecosystem of non-NVIDIA AI inference hardware.
Hugging Face and AMD Partner to Accelerate Models on CPU and GPU Platforms
Hugging Face and AMD announced a partnership aimed at optimizing and accelerating state-of-the-art AI models across AMD's CPU and GPU hardware platforms. The collaboration targets improved performance for models hosted and distributed through Hugging Face's ecosystem. This represents a strategic move to broaden hardware support beyond NVIDIA-dominated infrastructure in the AI/ML deployment landscape.
Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU
Hugging Face demonstrates a method for running RLHF fine-tuning on 20-billion-parameter language models using a single 24GB consumer GPU by combining TRL and PEFT (parameter-efficient fine-tuning). The approach uses techniques like LoRA and quantization to dramatically reduce memory requirements. This lowers the hardware barrier for RLHF experimentation from multi-GPU server setups to consumer-grade hardware.