vllm-omni-d3c89643·2 events·first seen Aliases: vllm-omni
FlashRT is an agent harness that guides coding agents to transform developer-written reference implementations of real-time multimodal pipelines (voice agents, video generation, multimodal LLMs) into optimized multi-GPU deployments. Using a 'chain-of-program' paradigm, the system directs an agent through IR construction, static analysis, and a measurement-gated optimization loop to tune placement, streaming, and parallelism. Across benchmarks on NVIDIA B200 and AMD MI355X GPUs, FlashRT achieves up to ~70x latency reduction and 3.6x throughput improvement, and outperforms the expert vLLM-Omni implementation for Qwen3-Omni text-to-audio inference on AMD hardware. The result suggests agent-driven optimization may be especially valuable on platforms lacking mature expert tooling.
The vllm-project has published vllm-omni, a Python framework extending vLLM's inference capabilities to omni-modality models. The repository has accumulated ~4,956 GitHub stars. It represents an expansion of the vLLM ecosystem into multimodal inference serving.