kimi-k3-in-c

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
           --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

--- generated text ---
 Paris.",
+            "The Eiffel
----------------------
8 tokens in 261.5 s, 32.69 s/token average
PEAK RSS for the whole run: 8.24 GB

Slow, and answering correctly, in 8.24 GB, from a checkpoint of 1.56 TB. This particular batch command deliberately asks for a raw continuation. The official Kimi K3 checkpoint is also chat-capable; use the XTML REPL below when you want answers and multi-turn history. Give the same batch request more memory and the answer does not change, only the clock:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
           --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental

--- generated text ---
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci
----------------------
28 tokens in 299.3 s, 10.69 s/token average
PEAK RSS for the whole run: 127.92 GB

Every figure in this document comes from the measurement output in docs/data/.

The dense trunk stays in memory to whatever depth you choose and streams the rest; the 1.45 TB of routed experts are never resident, and are multiplied straight out of their packed 4-bit form. The consequence is that the same model runs in 8 GB and in 224 GB and produces byte-identical output at every budget between.

Four decisions about where bytes live take it from a cluster to a laptop, and the answer at the bottom is the same as the answer at the top:

Part II builds every box in both diagrams from scratch, one component at a time.

Contents

Part I: Getting started

• Requirements
• Quick start: clone, build and verify in about a minute, with no model
• Full setup: the whole path to generated text
• Usage
• Synopsis
• Prompt options
• Memory options
• Generation options
• Diagnostic options
• Exit codes
• Environment variables
• Worked examples
• Choosing a preset
• Reading the run report
• Common questions

Part II: How it works

• The problem: a model that does not fit
• The four reductions
• The machine, and what it assumes
• The codebase
• Three invariants
• 1. Reading a 1.56 TB checkpoint from its headers
• 2. The config reader that refuses to guess
• 3. The tokenizer, byte for byte
• [4. Reduction one: the experts a