English | 中文

[](https://docs.xllm-ai.com/) [](https://quay.io/repository/jd_xllm/xllm-ai?tab=tags) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/xLLM-AI/xllm)

News

• 2026-08-27: 🎉 We day-0 support the GLM-5.3-Flash model, please refer to the Deployment Document for deployment.
• 2026-07-06: 🎉 xLLM is officially donated to the OpenAtom Foundation!
• 2026-06-13: 🎉 We day-0 support the MiniMax-M3 model, please refer to the Deployment Document for deployment.

Overview

xLLM is an efficient LLM inference framework, specifically optimized for Chinese AI accelerators, enabling enterprise-grade deployment with enhanced efficiency and reduced cost.

Highlights

• Top-tier Performance: Delivers high-throughput, low-latency inference through many advanced features.
• Mainstream Hardware Support: Purpose-built and deeply optimized for Chinese AI accelerators.
• Service-Engine Decoupled Architecture: Service layer handles scheduling and availability; engine layer handles computation.
• Enterprise-grade Deployment: Battle-tested at scale across JD.com's core retail business.

Hardware Support

Getting Started

• Quick Start
• Launch xLLM
• Online Service
• Offline Inference
• Supported Models

Community & Support

Acknowledgment

This project was made possible thanks to the following open-source projects:
- ScaleLLM - xLLM draws inspiration from ScaleLLM's graph construction method and references its runtime execution. - Mooncake - Build xLLM hybrid KV cache management based on Mooncake. - brpc - Build high-performance http service based on brpc. - tokenizers-cpp - Build C++ tokenizer based on tokenizers-cpp. - safetensors - xLLM relies on the C binding safetensors capability. - Partial JSON Parser - Implement xLLM's C++ JSON parser with insights from Python and Go implementations. - concurrentqueue - A fast multi-producer, multi-consumer lock-free concurrent queue for C++11.

Thanks to the following collaborating university laboratories:

• THU-MIG (School of Software, BNRist, Tsinghua University)
• USTC-Cloudlab (Cloud Computing Lab, University of Science and Technology of China)
• Beihang-HiPO (Beihang HiPO research group)
• PKU-DS-LAB (Data Structure Laboratory, Peking University)
• PKU-NetSys-LAB (NetSys Lab, Peking University)
• TJU-TANKLab (TANK Lab, Tianjin University)

Thanks to all the following developers who have contributed to xLLM.

Citation

If you think this repository is helpful to you, welcome to cite us:

@article{liu2025xllm,
  title={xLLM Technical Report},
  author={xLLM team},
  journal={arXiv preprint arXiv:2510.14686},
  year={2025}
}