Sophgo

MLIR-based compilation method for LLM inference on specialized hardware

Researchers present an MLIR-based compiler pipeline for deploying large language models on AI accelerators, using two dialect layers (TopOp for framework-agnostic graph representation and TpuOp for hardware-specific lowering). The method splits each Transformer layer into three static compilation stages (prefill, prefill_kv, decode) to handle the distinct computational profiles of prompt processing and autoregressive generation. The approach is implemented in the open-source TPU-MLIR compiler and LLM-TPU project, supporting Qwen, Llama, InternVL, and MiniCPM-V families with GPTQ, AWQ, and AutoRound quantization.