📦pytorch / torchtitan
← 返回首页
A PyTorch native platform for training generative AI models
⚖️ BSD-3-Clause📅 创建于 2023-12-13🔄 更新于 2026-11-21
🔗 GitHub Python
本站解读

torchtitan 用纯 PyTorch 组件搭起一套大规模生成式模型训练流程,从预训练、监督微调一路走到强化学习,换阶段不用重写模型。并行策略做成可组合的:模型声明自己如何切分,FSDP2、TP、PP、CP、EP 通过 DeviceMesh 与 DTensor 叠成多维布局。代码库刻意保持精简、易改,每次训练都是一份 Python recipe,改一处配置就能重跑一轮实验;文档按快速开始、模型、特性、RL 分块组织,方便按阶段查。作者声明这是 clean-room 的 PyTorch 原生实现。BSD-3-Clause 许可,允许修改与商用,保留版权与免责声明即可。适合要研究并行策略、又不想被封闭训练框架绑住的团队。

上榜 2 天 · 首次上榜 2026-11-20 · 近 7 天 +7 star · 榜单最高 #13 · BSD-3-Clause

📄 仓库 README

在 GitHub 查看 ↗
来自 GitHub 仓库自述文件 · 版权归原作者所有
torchtitan **A PyTorch-native platform for pretraining and post-training generative AI models at scale** [![arXiv](https://img.shields.io/badge/arXiv-2410.06511-b31b1b.svg)](https://arxiv.org/abs/2410.06511) [![ICLR](https://img.shields.io/badge/ICLR-2025-violet.svg)](https://iclr.cc/virtual/2025/poster/29620) [![devlogs](https://img.shields.io/badge/PyTorch-DevLog-DE3412.svg)](https://docs.pytorch.org/devlogs/) [![license](https://img.shields.io/badge/license-BSD_3--Clause-lightgrey.svg)](https://github.com/pytorch/torchtitan/blob/HEAD/LICENSE) [![pip](https://img.shields.io/pypi/v/torchtitan?color=blue)](https://pypi.org/project/torchtitan/) [![conda](https://img.shields.io/conda/vn/conda-forge/torchtitan?color=green)](https://anaconda.org/conda-forge/torchtitan) [Quick start](#quick-start) · [Models](#supported-models) · [Features](#features) · [RL](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/README.md) · [Docs](#documentation) · [Contributing](#contributing)

torchtitan is a clean-room, PyTorch-native implementation of large-scale training. One readable codebase takes a model from pretraining through supervised fine-tuning to reinforcement learning, composing FSDP, tensor, pipeline, context, and expert parallelism without rewriting the model.

  • Composable parallelism. Models declare how they shard; FSDP2, TP, PP, CP, and EP stack into N-D layouts through DeviceMesh and DTensor.
  • Built for research. Minimal, hackable code; every run is a Python recipe you can read, copy, and modify.
  • Pretraining to RL in one stack. TitanRL runs the same model definitions and kernels in the trainer and in vLLM generation, with a bitwise-reproducible batch-invariant mode.
  • Frontier models and techniques. DeepSeek, Qwen, Kimi, gpt-oss, Llama, and FLUX, with MXFP8/NVFP4 low precision, Dist-MoE, CUDA graphs, and more.

[!NOTE] torchtitan is under active development. To use the latest features, use a recent PyTorch nightly.

Latest news

  • [2026/10] Zephon, a data loader by DatologyAI, released its TorchTitan integration. Zephon keeps the same training batches in the same order even when your GPU count changes (elastic determinism). Check out the blog post on pytorch.org.
  • [2026/08] TitanRL is a hackable RL stack for scaling and debugging. It reuses TorchTitan model definitions and kernels across training and vLLM generation and supports batch-invariant mode.
  • [2025/11] AMD released an optimized fork of torchtitan for AMD GPUs.
  • [2025/10] We released torchtitan v0.2.0.
  • [2025/10] SkyPilot now supports torchtitan! See the tutorial.
Older news - **[2025/07]** We published [instructions](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/models/README.md) on how to add a model to `torchtitan`. - **[2025/04]** Our paper was accepted by [ICLR 2025](https://iclr.cc/virtual/2025/poster/29620). - **[2024/12]** GPU MODE [lecture](https://www.youtube.com/watch?v=VYWRjcUqW6w) on torchtitan. - **[2024/07]** [Presentation](https://pytorch2024.sched.com/event/1fHn3) at PyTorch Conference 2024.

Quick start

1. Install with a PyTorch nightly (Python 3.11+). Replace cu132 with your CUDA or ROCm build.

git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txt

2. Train a debug model. It uses the tokenizer bundled with the tests, so no download is needed.

NGPU=1 ./run_train.sh   # Llama 3 debug model, 10 steps

3. Train a real model. Download the tokenizer (Llama weights require access on Hugging Face), then select a recipe by module and function name:

python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
MODULE=torchtitan_recipes.models.llama3 CONFIG=llama3_8b ./run_train.sh   # 8 GPUs

4. Make it yours. A recipe is a Python function returning a complete config, so changing a run means writing a function, not passing flags:

# my_recipes.py
from torchtitan_recipes.models.llama3 import llama3_8b

def llama3_8b_tp2():
    config = llama3_8b()
    config.parallelism.tensor_parallel_degree = 2
    return config
MODULE=my_recipes CONFIG=llama3_8b_tp2 ./run_train.sh

See the configuration guide for the full model, and torchtitan_recipes for the verified recipes.

Supported models

Model Sizes Modality Verified recipes RL
Llama 3 1B – 405B Text llama3_8b, llama3_70b ✅
Qwen3 0.6B – 32B dense; 30B-A3B, 235B-A22B MoE Text qwen3_14b ✅
Qwen3.5 / 3.6 / 3.8 0.8B – 397B-A17B; 2.4T-A95B (3.8) Text, vision ✅ text
DeepSeek V3 16B, 236B, 671B Text deepseek_v3_671b (+ Dist-MoE, NVFP4)
DeepSeek V4 Flash, Pro Text Soon
gpt-oss 20B, 120B Text ✅
Kimi K2.7 Moonlight-16B-A3B, Kimi-VL-A3B, K2.5 Text, vision
Kimi K3 K3 Text, vision Soon
Muse Glimmer 30B Text, vision muse_glimmer_30b ✅
Muse-Spark-Open 816B-A42B Text, vision, audio muse_spark_816b_a42b_dist_moe_bf16_spmd_256 ✅ text
FLUX dev, schnell Text-to-image flux_dev, flux_schnell

Every model has debug configurations exercised in CI. Verified recipes have been run in training, convergence, or performance workflows. To add a model, see the model guide.

Features

**Parallelism** - [FSDP2](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/distributed/FSDP.md), HSDP, and DDP - Tensor parallel, including async TP - [Pipeline parallel](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/distributed/PIPELINE_PARALLEL.md): 1F1B, Interleaved 1F1B, GPipe, and other `torch.distributed.pipelining` schedules - Context parallel: all-gather KV and Ulysses - Expert parallel with DeepEP / HybridEP dispatch - [Dist-MoE](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/models/common/dist_moe/README.md): fused dispatch, expert compute, and combine on Blackwell **Memory and performance** - [Activation checkpointing](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/distributed/ACTIVATION_REMAT.md): full, selective, and regional - [Activation offloading](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/distributed/ACTIVATION_OFFLOAD.md) to CPU of the activations remat (activation checkpointing) retains - Regional `torch.compile` and CUDA graphs - [MXFP8](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/quantization/mxfp8/README.md) and [NVFP4](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/quantization/nvfp4/README.md) training - FlashAttention 3/4, FlexAttention, and varlen attention - [BF16 optimizer states](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/components/optim/BF16_OPTIMIZER_STATES.md) and FSDP CPU offload **Training** - [Grain data pipeline](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/components/data/README.md) with packing, mixing, and checkpointable state - Pretraining, SFT on chat datasets, and LoRA - Muon (DistMuon), EMA, multi-token prediction - Warmup-stable-decay LR schedule and [validation](https://github.com/pytorch/torchtitan/blob/HEAD/docs/evaluation.md) **Post-training with [TitanRL](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/README.md)** - One model definition shared by the trainer and vLLM - [Bitwise trainer/generator parity](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/BITWISE_PARITY.md) in batch-invariant mode - Async rollouts orchestrated with Monarch; trainer-to-generator model state dict transfer over TorchStore - GRPO and DAPO losses; [DAPO Math](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/examples/dapo_math), [Search-R1](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/examples/search_r1), and [Verifiers](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/examples/verifiers) examples **Checkpointing** - [Distributed checkpointing](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/components/checkpointer/README.md), including async saves - Load and save Hugging Face safetensors directly - [Conversion scripts](https://github.com/pytorch/torchtitan/blob/HEAD/scripts/checkpoint_conversion) between HF and DCP **Debugging and observability** - [Metrics](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/observability/METRICS.md): loss, memory, throughput, TFLOPs, and MFU in TensorBoard or W&B - [Profiling and memory snapshots](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/observability/README.md) - [Flight Recorder and other debugging tools](https://github.com/pytorch/torchtitan/blob/HEAD/docs/debugging.md) - Fake-backend dry runs of large layouts on one GPU - Deterministic mode and [loss comparison](https://github.com/pytorch/torchtitan/blob/HEAD/scripts/loss_compare.py) across commits - [SDC replay](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/observability/SILENT_DATA_CORRUPTION.md) and [structured logging](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/observability/structured_logger/README.md) **Extensibility** - Python [recipes](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/config/README.md) selected with `--module` and `--config` - [Component overrides](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/config/OVERRIDE.md) for custom kernels and modules - [Experiments](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/experiments): GraphTrainer, TorchFT, Transformers modeling backend

We report performance on up to 512 GPUs and verify loss convergence across parallelisms and techniques.

Test status

Hardware Integration tests Unit tests
CPU CPU Unit Test
NVIDIA GPU Integration Tests H100 Tests B200 Tests GPU Unit Tests
AMD GPU (ROCm) Integration Tests (ROCm)
RL (NVIDIA GPU) RL Integration Tests RL B200 Tests RL GPU Unit Tests

Installation

The quick start installs from source, which is the recommended way to get the latest features. Other options:

Nightly builds
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132 --force-reinstall
pip install --pre torchtitan --index-url https://download.pytorch.org/whl/nightly/cu132
Replace `cu132` with another CUDA version or a ROCm build (e.g. `rocm10.0`).
Stable releases
pip install torchtitan
# or
conda install conda-forge::torchtitan
Each stable release pins the nightly versions of `torch` and `torchao` it was validated against; see [RELEASE.md](https://github.com/pytorch/torchtitan/blob/HEAD/RELEASE.md).
Optional dependencies - **Low precision (MXFP8, NVFP4):** install the `torchao` nightly CI pins, from the index matching your `torch` build. It is not in `requirements.txt` so it cannot be resolved against a different `torch`: ```bash USE_CPP=0 pip install --pre -r .github/requirements-torchao.txt --index-url https://download.pytorch.org/whl/nightly/cu132 ``` - **RL:** `scripts/install_rl_env.sh` installs the vLLM, Monarch, and TorchStore nightlies and the FlashAttention build for your GPU. See the [RL quick start](https://github.com/pytorch/torchtitan/blob/HEAD/torchtitan/rl/README.md#quick-start). - **Extras** for vision, FLUX, RL, and development are also available as `pip install -e ".[vlm,flux,rl,dev]"`. - **Importing from elsewhere:** the source tree runs as is; to import `torchtitan` from another directory, install it in editable mode without re-resolving dependencies: `pip install -e . --no-deps`.
Multi-node training with Slurm Submit [`scripts/multinode_trainer.slurm`](https://github.com/pytorch/torchtitan/blob/HEAD/scripts/multinode_trainer.slurm) with `sbatch` from the repository root. Set the node count in both the `#SBATCH` header and the `torchrun --nnodes` argument:
#SBATCH --ntasks=2
#SBATCH --nodes=2

srun torchrun --nnodes 2 ...
If a node does not have 8 GPUs, also adjust `--nproc_per_node` and `#SBATCH --gpus-per-task`.

Documentation

Topic Where
Configuring runs and writing recipes torchtitan/config
Adding a model torchtitan/models
Parallelisms and distributed techniques torchtitan/distributed
Datasets and data loading torchtitan/components/data
Checkpointing and HF conversion torchtitan/components/checkpointer
Reinforcement learning torchtitan/rl
Debugging, reproducibility, and validating numerics docs/debugging.md
Validation and evaluation docs/evaluation.md
Benchmarks docs/benchmarks
Tests and CI tests

Contributing

We look forward to your contributions!

Citation

Our paper gives a detailed look at the parallelisms and optimizations in torchtitan, with advice on when to use each technique: TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training.

@inproceedings{
   liang2025torchtitan,
   title={TorchTitan: One-stop PyTorch native solution for production ready {LLM} pretraining},
   author={Wanchao Liang and Tianyu Liu and Less Wright and Will Constable and Andrew Gu and Chien-Chin Huang and Iris Zhang and Wei Feng and Howard Huang and Junjie Wang and Sanket Purandare and Gokul Nadathur and Stratos Idreos},
   booktitle={The Thirteenth International Conference on Learning Representations},
   year={2025},
   url={https://openreview.net/forum?id=SFN6Wm7YBI}
}

License

Source code is made available under a BSD 3 license. You may have other legal obligations that govern your use of other content linked in this repository, such as the license or terms of service for third-party data and models.

📈 近 7 天 Star 趋势

每日新增 Star(柱)与累计 Star(线)· 数据来自 GitHub

🏆 排名变化趋势

在各榜单中的排名历史(数字越小越靠前)

📋 历史上榜记录

每次快照的详细数据
日期 榜单 排名 周期新增 总 STAR