DeepSeek V4 Flash Vision-Exp DSpark on 2x DGX Spark

_{by Mia'a AI Lab}

Two-node DGX Spark recipe for deepseek-ai/DeepSeek-V4-Flash-Vision-Exp: vLLM TP=2, DSpark speculative decoding, 1M-token ceiling, nvfp4_ds_mla KV. Native image support is a startup hotfix on the Anemll 0.1.1 runtime (ViT + Aligner from the Vision-Exp checkpoint, OpenAI image_url / <image>path</image>). There is no video encoder in the official weights; GIF is a still frame. The old Qwen3-VL sidecar / MCP path is removed.

The files added in 7f1217f, including the files under recipe/overlay/, came from Tony D (tonyd2wild). He is the author of that work.

Default image: ghcr.io/anemll/dspark-vllm-gx10:0.1.1

Numbers: results/RESULTS-2026-08-14.md (dated tables, method, historical lanes). Checkpoint / encoder: docs/DEEPSEEK_V4_FLASH_0731.md.

---

Quick start

Run everything from the head node. You need two DGX Sparks, RoCE/NCCL working, and the same image on both. By default each node has its own HuggingFace checkpoint (prepare copies onto the worker). Set DSPARK_WORKER_HF_NFS=1 to keep weights only on the head: the worker mounts that cache over NFSv4 on the ConnectX link (same pattern as Qwen3.8-Flash-vLLM).

1. Env

bash cp .env.dspark.example .env.dspark

Set at least: WORKER_HOST, MASTER_ADDR, NCCL_IB_HCA, NCCL_SOCKET_IFNAME (and matching TP_ / GLOO_ IF names), VLLM_HOST_IP, WORKER_VLLM_HOST_IP, HF_CACHE. WORKER_HF_CACHE is the worker checkpoint path. When DSPARK_WORKER_HF_NFS=1, it is the worker's local JIT overlay on the NFS mount. If the worker checkout is not the same path, set WORKER_DIR / WORKER_SCRIPT_DIR.

Example fabric (edit f0 vs f1 and GID for your ring):

env WORKER_HOST=10.0.0.2 MASTER_ADDR=10.0.0.1 VLLM_HOST_IP=10.0.0.1 WORKER_VLLM_HOST_IP=10.0.0.2 NCCL_IB_HCA=rocep1s0f1 NCCL_SOCKET_IFNAME=enp1s0f1np1 DSPARK_VLLM_IMAGE=ghcr.io/anemll/dspark-vllm-gx10:0.1.1

Leave serving knobs at the defaults unless you mean to change them. Meaningful on/off flags (ABLITERATED, thinking, hotfixes) are listed under .env.dspark switches.

2. Image on both nodes

bash docker pull ghcr.io/anemll/dspark-vllm-gx10:0.1.1

Repeat on the worker (or pull there via ssh). Start refuses to launch if either node is missing the image.

3. Weights on the head

bash ./prepare-dspark-model-cache.sh --official

Use --abliterated or --yes (reads ABLITERATED from .env.dspark). Abliteration is gated (HF_TOKEN): agree on the Keys Hub repo, then prepare downloads the 18 KiB direction — not the 157 GiB checkpoint. Prepare forces HF online even if HF_HUB_OFFLINE=1, then you can serve offline. Default DSPARK_WORKER_HF_NFS=0 also downloads onto the worker. After the cache is complete, keep HF_HUB_OFFLINE=1. See Worker weights over NFS to skip the second copy.

4. Optional CPU gates (no GPU; will not measure tok/s)

bash bash scripts/ci-validate.sh

5. Start (worker first, then head)

bash ./start-deepseek-v4-flash-dspark.sh

One-shot bind override: ./start-deepseek-v4-flash-dspark.sh --host 0.0.0.0 --port 9000. After a reboot, dockerd may already have restored the ranks (restart: unless-stopped); start then exits 3 (already running), not 1. That is expected — do not ./stop unless you want a cold start. systemd: SuccessExitStatus=3.

Optional three Sparks (TP=3) is a separate launcher so .env cannot flip the 2-node path: ./start-tp3.sh (needs WORKER2_HOST; see Optional: three Sparks (TP=3) and docs/TP3.md).

6. Check it is up

bash curl -fsS http://127.0.0.1:8888/v1/models ./smoke-deepseek-v4-flash-dspark.sh ./status-deepseek-v4-flash-dspark.sh

Expect "id": "deepseek-v4-flash-vision-exp" and "max_model_len": 1048576. Boot log (trust the live numbers):

text Available KV cache memory: 17.04 GiB GPU KV cache size: 2,331,430 tokens Maximum concurrency for 1,048,576 tokens per request: 2.22x

API: http://HEAD_NODE_IP:8888/v1 (VLLM_HOST=0.0.0.0 by default). Head-only tests: VLLM_HOST=127.0.0.1.

Day-to-day: ./status-…, ./logs-…, ./stop-…. Disable earlyoom on both hosts or it can kill vLLM under deep-context load.

---

Default profile

Knob	Default
Image	ghcr.io/anemll/dspark-vllm-gx10:0.1.1
Checkpoint	official Vision-Exp @ 86f746b36186f0e567729a5c06a8c918caba82a9 (ABLITERATED=0)
Served name	deepseek-v4-flash-vision-exp
Context ceiling	MAX_MODEL_LEN=1048576 (1M)
Concurrent seqs	MAX_NUM_SEQS=6
Batch tokens	MAX_NUM_BATCHED_TOKENS=8192
KV	nvfp4_ds_mla, 17.04 GiB / 2,331,430 tokens on this cluster (util 0.835; Vision-Exp ViT takes more weight RAM than 0731)
Spec	MTP_NUM_TOKENS=6 (≥ dspark_block_size 5 and divisible by Vision-Exp n_predict=3)
Thinking	DEFAULT_THINKING=low (off / low / high / max)
Graphs	VLLM_USE_BREAKABLE_CUDAGRAPH=0 (keep this; unset is slower)

start-*.sh exports GPU_MEMORY_UTILIZATION from GPU_MEMORY_UTILIZATION_TEXT. Do not set GPU_MEMORY_UTILIZATION by hand.

max_model_len and max_num_seqs are ceilings, not reservations. The limit is sum(live tokens) ≤ KV pool. Six normal agent turns fit; six simultaneous full-1M requests do not. See How the KV cache works.

Long coding / big prompts (optional, still 1M ceiling):

MAX_NUM_SEQS=4
MAX_NUM_BATCHED_TOKENS=16384
GPU_MEMORY_UTILIZATION_TEXT=0.87

---

.env.dspark switches

Copy .env.dspark.example → .env.dspark. Start syncs it to the worker. Restart both ranks after a flip (./stop-… then ./start-…). Do not set DSPARK_MODEL or GPU_MEMORY_UTILIZATION by hand.

NCCL/RoCE, CUDA arch, and compile knobs stay in the example file — they are cluster wiring, not product switches. Full Anemll vs Stage-C matrix: docs/ENVS.md.

Weights

Variable	Default	What it does
ABLITERATED	0	0 = official deepseek-ai/DeepSeek-V4-Flash-Vision-Exp @ DSPARK_REVISION, stock decoder. 1 = same official weights plus runtime refusal-direction projection. Does not download the 157 GiB Keys checkpoint. You must agree to that repo's gated terms, then prepare --abliterated downloads RESPONSIBLE_USE.md plus the 18 KiB direction (HF_TOKEN). Recreate both ranks after flipping. Default λ=3.5, layers 10-42. The direction was captured on 0731 FP8 DSpark; transfer onto Vision-Exp is experimental.
DSPARK_REVISION	86f746b36186f0e567729a5c06a8c918caba82a9	Official Vision-Exp pin. Empty = tip of main.
DSPARK_REVISION_ABLITERATED	empty	Abliterated pin. Empty = tip of that repo.
DSPARK_MODEL_OFFICIAL / DSPARK_MODEL_ABLITERATED	the two HF ids above	Override only if you intentionally swap the repo id. Do not point this at the 0731 ablit dump — that drops image_url.
SERVED_MODEL_NAME	deepseek-v4-flash-vision-exp	Space-separated aliases; clients may send any alias as model. Startup probes, warmup, and smoke use the first alias.
HF_HUB_OFFLINE	1	1 after the hub cache is warm. Prepare forces online for the download.
DSPARK_WORKER_HF_NFS	0	0 (default) = bind WORKER_HF_CACHE as a second copy (prepare downloads on the worker). 1 = worker mounts head HF_CACHE over NFSv4 on ConnectX (no local checkpoint).

Flip ABLITERATED like this:

# in .env.dspark
ABLITERATED=1

./prepare-dspark-model-cache.sh --yes    # or --abliterated / --official
./stop-deepseek-v4-flash-dspark.sh
./start-deepseek-v4-flash-dspark.sh

--official writes ABLITERATED=0; --abliterated writes 1.

Worker weights over NFS (optional)

Default is off: DSPARK_WORKER_HF_NFS=0. prepare downloads the checkpoint onto the worker as well.

Set DSPARK_WORKER_HF_NFS=1 in .env.dspark to skip that second Hub download. Start then exports the head HF_CACHE via NFSv4 on NCCL_SOCKET_IFNAME (ConnectX). A live exporter on that address is reused (for example Qwen's vllm-fn-nfs); otherwise start brings up dspark-nfs. The worker Docker volume dspark-hf mounts the share read-only. Triton, TileLang, vLLM, FlashInfer, CuTe, and NCCL-FR caches stay on the worker host as overlays under WORKER_HF_CACHE. ./stop-deepseek-v4-flash-dspark.sh --nfs tears down only dspark-nfs, not Qwen's share.

DSPARK_WORKER_HF_NFS=1
# NFS_SERVER_IP=10.0.22.1   # optional; default is IPv4 on NCCL_SOCKET_IFNAME

If official Vision-Exp is already cached, overlay the 26 edited shards (~87 GiB) instead of re-fetching the full ~157 GiB dump:

python3 scripts/overlay-vision-exp-ablit-cache.py

The 0731 abliterated freeze stays on branch 0731-ablit.

Images: OpenAI image_url (JPEG/PNG/GIF/WebP; GIF is still-frame).