The SOTA tokenization library on all languages, all models, all hardwares. Our goal with `tokenizers` is to develop and maintain the industry's standard tokenization engine, making it the defacto place for everyone to contribute to the whole ecosystem. ### Release candidate: v1.0.0.rc.x As we are switching from 0.23 to v1.0.0, the current library does not ship all features. If you are worried, check the [v1 features](#v1-features) section for more details of what we are bringing back. We will publish a blog about what breaking changes we introduced. We strived to keep them as small as possible. # Installation

pip install --pre tokenizers

For Rust, add the crate to your `Cargo.toml`:

[dependencies]
tokenizers = "1.0.0"

Training (`train`) and downloading from the Hub (`http`) are opt-in features. The crate's [Using the crate](https://github.com/huggingface/tokenizers/blob/HEAD/tokenizers/README.md#using-the-crate) section lists every feature and how to build a smaller binary. # Usage

>>> from tokenizers import Tokenizer
>>> tokenizer = Tokenizer.from_pretrained("meta-llama/Llama-3.1-8B")   # or .from_file(path)
>>> tokenizer.encode("Hello, y'all! How are you 😁 ?")
Encoding(ids=[128000, 9906, 11, 379, 65948, 0, 2650, 527, 499, 27623, 223, 949], type_ids=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], attention_mask=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])
>>> tokenizer.tokenize("Hello")
['<|begin_of_text|>', 'Hello']
>>> tokenizer.decode([15339, 1917])
'hello world'
>>> tokenizer.decode_tokens(tokenizer.encode("Hello"))
['<|begin_of_text|>', 'Hello']

# Performances For more details, checkout https://huggingface-tokenizers-v1.static.hf.space/index.html. # Bindings | | status | |---|---| | [Rust](https://github.com/huggingface/tokenizers/blob/HEAD/tokenizers) | ✅ reference implementation | | [Python](https://github.com/huggingface/tokenizers/blob/HEAD/bindings/python) | ✅ | | [Node.js](https://github.com/huggingface/tokenizers/blob/HEAD/bindings/node) | ✅ | | C / C++ / Java / Go/ etc | 🚧 planned | | [Ruby](https://github.com/ankane/tokenizers-ruby) | community, external repo | # v1.0.0 roadmap: As we work toward v1, we are gonna bring back the entire python API that allows `transformers`' style of interacting with a tokenizer object: ## TODOS: ### Python Bindings - Bring training back: this should be easy, but we want the perfs gains to come as well. - Batch encoding API with zero-copy views on encoded ids - dlpack API for torch / JAX / numpy interop. - Composition: edit components of the pipeline (`tokenizer.normalizer = [...]`). - `tokenizer.save` to persist a modified or trained tokenizer on disk. - `role_to_token` override. - Accept pair input for `encode` / `encode_batch`. - Bring back offset output: this should be useful for people training. - Windowing / stride in encode. - `decode_stream`. ### Rust core - `bitnorm`: more performance from optimized normalization, as this will bring breaking changes we want to make sure it gets to v1. - Windowing (truncation with stride). - Improve training performance - Improve decode performance - Offset tracking - Padding: 2D flat buffer - Wordcache: `shrn` vs emulated movemask - Fix ModernBERT / OLMo load failure ("64-bit hash collision"). - Unroll more regex: we need to cover the most used ones, we might have missed one or two. ### Other - C & C++ bindings for loading, encode and decode APIs ## After 1.0.0 - Improve / rewrite the `bitcannon` crate - Java, Go, Swift... bindings, supported by the C bindings - `tk-devices` — exploratory GPU encoding and batch decoding that keeps text and ids on the device: upload the vocabulary once, compute output positions in parallel, gather the bytes on the GPU. An optional component aimed at large batches, subject to prototyping and measurement. # Crate details ## Crate size ## Sub-crates

tk-encode — the inference engine

The models (BPE, Unigram, WordPiece, WordLevel) and the full pipeline: `Normalizer`, `PreTokenizer`, `Model`, `PostProcessor`, `Decoder`. **Why separate:** you don't need all features to serve a model in production settings. Splitting it from training means a serving binary never links a trainer, a corpus reader, legacy converters, etc.

bitcannon — bitstream pre-tokenization

Unicode tag classification plus pre-tokenization as a **bitstream program**. Follows *Interleaved Bitstream Execution for Multi-Pattern Regex Matching on GPUs* (MICRO'25, [10.1145/3725843.3756052](https://doi.org/10.1145/3725843.3756052)) but we added our own algorithm to make sure we don't need a parabix engine. Characters are first classified into "tags", which use a sparse table generated when compiling `bitcannon`. This means there are 0 depencencies to the heavy unicode tables. Ships byte-exact grammars for gpt2 / ByteLevel, cl100k, o200k, tekken, deepseek and kimi-k2. The classes or tags each have a high and low nibble (encoded on a u8), and carry refinement when needed. Using u8 tags allows us to leverage less bitsream, meaning have a simpler / faster pre tokenization. **Why separate:** we just thought it might be useful to some people someday. As far as we know, this implementation is the fastest for every single pretokenizer regex on CPU.

tk-serialize — the reader

`from_json_file` turns a canonical `tokenizer.json` into a `PipelineTokenizer`. **Why separate:** this just makes it optional for some usecases!

tk-convert — the upgrade pass

`canonicalize_file` rewrites legacy a `tokenizer.json` written by any older version of this library into the canonical form the reader accepts. A pure JSON→JSON rewrite; it depends on nothing but `std::path` and `serde_json`. **Why separate:** every config ever published stays readable without the runtime carrying a decade of compatibility branches. But, the legacy code amounted to quite a lot of the final crate size. Disabling it allows on device builds to be smaller.

tk-train — the training half

The `Trainer` trait, every concrete `*Trainer`, `TrainerWrapper` and the `Trainable` extension. The `tokenizers` crate re-exports it behind the `train` feature, which is off by default. It trains only the models whose feature is on (`bpe`, `unigram`, `wordpiece`, `wordlevel`). Add `esaxx_fast` for the C++ suffix array behind Unigram training. **Why separate:** training is a batch job on a workstation; inference is a hot loop in a server. They have opposite constraints, so they get opposite dependency budgets.

bitmap_gen — dev-only table generator

`cargo run -p bitmap_gen` regenerates `bitcannon`'s committed classify tables from `unicode-properties`, emitting one `Atom` tag per codepoint. **Why separate — this is what buys the zero Unicode dependency.** `unicode-properties` is a dependency of *this* crate and of nothing else: the tables it produces are checked into `bitcannon/src/classify/atom_tables.rs` as ordinary source, so the Unicode data is resolved once, at development time, by a crate that is never linked into anything that ships and is never published. No build script either — `bitcannon` compiles with no code generation step, and a release binary contains the tags without containing a Unicode crate to derive them. The release workflow re-runs the generator and fails if the committed table differs, so "baked" cannot silently mean "stale".

tokenizers — umbrella

A thin re-export so existing `tokenizers::…` paths keep working. Depend on this one unless you know you want less.

## Hardware **No SIMD is required.** Every kernel has a portable path that is always compiled and is the byte-exact test oracle for its vectorised siblings, so correctness never depends on a kernel being present — only throughput does.

Which ops are hardware-adapted

| op | aarch64 | x86_64 | wasm32 | portable fallback | |---|---|---|---|---| | Unicode atom classify | NEON (baseline) | AVX-512 VBMI → SSE4.1/SSSE3, runtime-detected | SIMD128 | `classify_scalar` | | bitstream block build | NEON | SSE/AVX | — | `build_block_scalar` | | literal & added-token scan | NEON | x86_64 | — | scalar | | vocab bucket nibble match | NEON | — | — | scalar | aarch64 selects at compile time (NEON is baseline). x86_64 dispatches at runtime via `is_x86_feature_detected!`, so one binary covers every x86_64 CPU since 2008. wasm32 needs the `simd128` target feature; without it, the scalar walk.

# Acknowledgements This work stands on the shoulders of a lot of open source projects. Thanks to all of them for their great work. [gigatoken](https://github.com/marcelroed/gigatoken), [tiktoken](https://crates.io/crates/tiktoken-rs), [kitoken](https://crates.io/crates/kitoken), [tokie](https://crates.io/crates/tokie), [fastokens](https://crates.io/crates/fastokens), [wordchipper](https://crates.io/crates/wordchipper) and [ai-tokenizer](https://www.npmjs.com/package/ai-tokenizer) each pushed on what a fast tokenizer can be; we read that work, and several of the ideas above reached us because another project showed they were worth trying. Docs: [guide](https://huggingface.co/docs/tokenizers/index) · [quicktour](https://huggingface.co/docs/tokenizers/quicktour) · [docs.rs](https://docs.rs/tokenizers/)