torchtitan 用纯 PyTorch 组件搭起一套大规模生成式模型训练流程,从预训练、监督微调一路走到强化学习,换阶段不用重写模型。并行策略做成可组合的:模型声明自己如何切分,FSDP2、TP、PP、CP、EP 通过 DeviceMesh 与 DTensor 叠成多维布局。代码库刻意保持精简、易改,每次训练都是一份 Python recipe,改一处配置就能重跑一轮实验;文档按快速开始、模型、特性、RL 分块组织,方便按阶段查。作者声明这是 clean-room 的 PyTorch 原生实现。BSD-3-Clause 许可,允许修改与商用,保留版权与免责声明即可。适合要研究并行策略、又不想被封闭训练框架绑住的团队。
torchtitan is a clean-room, PyTorch-native implementation of large-scale
training. One readable codebase takes a model from pretraining through
supervised fine-tuning to reinforcement learning, composing FSDP, tensor,
pipeline, context, and expert parallelism without rewriting the model.
DeviceMesh and DTensor.[!NOTE]
torchtitanis under active development. To use the latest features, use a recent PyTorch nightly.
torchtitan for AMD GPUs.torchtitan v0.2.0.torchtitan! See the tutorial.1. Install with a PyTorch nightly (Python 3.11+). Replace cu132 with your CUDA or ROCm build.
git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txt
2. Train a debug model. It uses the tokenizer bundled with the tests, so no download is needed.
NGPU=1 ./run_train.sh # Llama 3 debug model, 10 steps
3. Train a real model. Download the tokenizer (Llama weights require access on Hugging Face), then select a recipe by module and function name:
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
MODULE=torchtitan_recipes.models.llama3 CONFIG=llama3_8b ./run_train.sh # 8 GPUs
4. Make it yours. A recipe is a Python function returning a complete config, so changing a run means writing a function, not passing flags:
# my_recipes.py
from torchtitan_recipes.models.llama3 import llama3_8b
def llama3_8b_tp2():
config = llama3_8b()
config.parallelism.tensor_parallel_degree = 2
return config
MODULE=my_recipes CONFIG=llama3_8b_tp2 ./run_train.sh
See the configuration guide for the full model,
and torchtitan_recipes for the verified recipes.
| Model | Sizes | Modality | Verified recipes | RL |
|---|---|---|---|---|
| Llama 3 | 1B – 405B | Text | llama3_8b, llama3_70b |
✅ |
| Qwen3 | 0.6B – 32B dense; 30B-A3B, 235B-A22B MoE | Text | qwen3_14b |
✅ |
| Qwen3.5 / 3.6 / 3.8 | 0.8B – 397B-A17B; 2.4T-A95B (3.8) | Text, vision | ✅ text | |
| DeepSeek V3 | 16B, 236B, 671B | Text | deepseek_v3_671b (+ Dist-MoE, NVFP4) |
|
| DeepSeek V4 | Flash, Pro | Text | Soon | |
| gpt-oss | 20B, 120B | Text | ✅ | |
| Kimi K2.7 | Moonlight-16B-A3B, Kimi-VL-A3B, K2.5 | Text, vision | ||
| Kimi K3 | K3 | Text, vision | Soon | |
| Muse Glimmer | 30B | Text, vision | muse_glimmer_30b |
✅ |
| Muse-Spark-Open | 816B-A42B | Text, vision, audio | muse_spark_816b_a42b_dist_moe_bf16_spmd_256 |
✅ text |
| FLUX | dev, schnell | Text-to-image | flux_dev, flux_schnell |
Every model has debug configurations exercised in CI. Verified recipes have been run in training, convergence, or performance workflows. To add a model, see the model guide.
| **Parallelism** - [FSDP2](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/distributed/FSDP.md), HSDP, and DDP - Tensor parallel, including async TP - [Pipeline parallel](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/distributed/PIPELINE_PARALLEL.md): 1F1B, Interleaved 1F1B, GPipe, and other `torch.distributed.pipelining` schedules - Context parallel: all-gather KV and Ulysses - Expert parallel with DeepEP / HybridEP dispatch - [Dist-MoE](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/models/common/dist_moe/README.md): fused dispatch, expert compute, and combine on Blackwell **Memory and performance** - [Activation checkpointing](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/distributed/ACTIVATION_REMAT.md): full, selective, and regional - [Activation offloading](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/distributed/ACTIVATION_OFFLOAD.md) to CPU of the activations remat (activation checkpointing) retains - Regional `torch.compile` and CUDA graphs - [MXFP8](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/quantization/mxfp8/README.md) and [NVFP4](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/quantization/nvfp4/README.md) training - FlashAttention 3/4, FlexAttention, and varlen attention - [BF16 optimizer states](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/components/optim/BF16_OPTIMIZER_STATES.md) and FSDP CPU offload **Training** - [Grain data pipeline](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/components/data/README.md) with packing, mixing, and checkpointable state - Pretraining, SFT on chat datasets, and LoRA - Muon (DistMuon), EMA, multi-token prediction - Warmup-stable-decay LR schedule and [validation](https://github.com/pytorch/torchtitan/blob/HEAD/docs/evaluation.md) | **Post-training with [TitanRL](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/README.md)** - One model definition shared by the trainer and vLLM - [Bitwise trainer/generator parity](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/BITWISE_PARITY.md) in batch-invariant mode - Async rollouts orchestrated with Monarch; trainer-to-generator model state dict transfer over TorchStore - GRPO and DAPO losses; [DAPO Math](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/examples/dapo_math), [Search-R1](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/examples/search_r1), and [Verifiers](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/examples/verifiers) examples **Checkpointing** - [Distributed checkpointing](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/components/checkpointer/README.md), including async saves - Load and save Hugging Face safetensors directly - [Conversion scripts](https://github.com/pytorch/torchtitan/blob/HEAD/scripts/checkpoint_conversion) between HF and DCP **Debugging and observability** - [Metrics](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/observability/METRICS.md): loss, memory, throughput, TFLOPs, and MFU in TensorBoard or W&B - [Profiling and memory snapshots](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/observability/README.md) - [Flight Recorder and other debugging tools](https://github.com/pytorch/torchtitan/blob/HEAD/docs/debugging.md) - Fake-backend dry runs of large layouts on one GPU - Deterministic mode and [loss comparison](https://github.com/pytorch/torchtitan/blob/HEAD/scripts/loss_compare.py) across commits - [SDC replay](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/observability/SILENT_DATA_CORRUPTION.md) and [structured logging](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/observability/structured_logger/README.md) **Extensibility** - Python [recipes](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/config/README.md) selected with `--module` and `--config` - [Component overrides](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/config/OVERRIDE.md) for custom kernels and modules - [Experiments](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/experiments): GraphTrainer, TorchFT, Transformers modeling backend |
We report performance on up to 512 GPUs and verify loss convergence across parallelisms and techniques.
| Hardware | Integration tests | Unit tests |
|---|---|---|
| CPU | ||
| NVIDIA GPU | ||
| AMD GPU (ROCm) | ||
| RL (NVIDIA GPU) |
The quick start installs from source, which is the recommended way to get the latest features. Other options:
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132 --force-reinstall
pip install --pre torchtitan --index-url https://download.pytorch.org/whl/nightly/cu132
Replace `cu132` with another CUDA version or a ROCm build (e.g. `rocm10.0`).
pip install torchtitan
# or
conda install conda-forge::torchtitan
Each stable release pins the nightly versions of `torch` and `torchao` it was
validated against; see [RELEASE.md](https://github.com/pytorch/torchtitan/blob/HEAD/RELEASE.md).
#SBATCH --ntasks=2
#SBATCH --nodes=2
srun torchrun --nnodes 2 ...
If a node does not have 8 GPUs, also adjust `--nproc_per_node` and
`#SBATCH --gpus-per-task`.
| Topic | Where |
|---|---|
| Configuring runs and writing recipes | torchtitan/config |
| Adding a model | torchtitan/models |
| Parallelisms and distributed techniques | torchtitan/distributed |
| Datasets and data loading | torchtitan/components/data |
| Checkpointing and HF conversion | torchtitan/components/checkpointer |
| Reinforcement learning | torchtitan/rl |
| Debugging, reproducibility, and validating numerics | docs/debugging.md |
| Validation and evaluation | docs/evaluation.md |
| Benchmarks | docs/benchmarks |
| Tests and CI | tests |
We look forward to your contributions!
experiments folder;
follow the experiments guidelines.Our paper gives a detailed look at the parallelisms and optimizations in
torchtitan, with advice on when to use each technique:
TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training.
@inproceedings{
liang2025torchtitan,
title={TorchTitan: One-stop PyTorch native solution for production ready {LLM} pretraining},
author={Wanchao Liang and Tianyu Liu and Less Wright and Will Constable and Andrew Gu and Chien-Chin Huang and Iris Zhang and Wei Feng and Howard Huang and Junjie Wang and Sanket Purandare and Gokul Nadathur and Stratos Idreos},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=SFN6Wm7YBI}
}
Source code is made available under a BSD 3 license. You may have other legal obligations that govern your use of other content linked in this repository, such as the license or terms of service for third-party data and models.
| 日期 | 榜单 | 排名 | 周期新增 | 总 STAR |
|---|