vLLM Docker Optimized for DGX Spark (single or multi-node)

This repository contains the Docker configuration and startup scripts to run vLLM on DGX Spark, from a single node to multi-node clusters using Ray or vLLM's native PyTorch distributed mode. It supports InfiniBand/RDMA (NCCL), custom environment configuration, and high-performance model loading through fastsafetensors and InstantTensor. Cluster setup supports direct connections between dual Sparks, QSFP/RoCE switch configurations, 3-node mesh configurations, and closed rings.

While it was primarily developed to support multi-node inference, it works just as well on single-node setups.

If you point an AI agent at this repository to prepare a host or run a recipe, use the operational agent runbook. Agent-assisted repository changes use the separate development guide; AGENTS.md routes agents to the appropriate guide.

Table of Contents

• DISCLAIMER
• QUICK START
• REGULAR QUICK START
• CHANGELOG
• 1. Building the Docker Image
• 2. Launching the Cluster (Recommended)
• 3. Running the Container (Manual)
• 4. Configuration Details
• 5. Mods and Patches
• 6. Launch Scripts
• 7. Using cluster mode for inference
• 8. Model Loading
• 9. Benchmarking
• 10. Downloading Models

DISCLAIMER

This repository is not affiliated with NVIDIA or their subsidiaries. This is a community effort aimed to help DGX Spark users to set up and run the most recent versions of vLLM on Spark cluster or single nodes.

By default, build-and-copy.sh pulls the tested nightly runner image from DockerHub: eugr/spark-vllm:latest. Nightly images are built and tested on multiple models in both cluster and solo configuration before latest is advanced. We will expand the selection of models we test in the pipeline, but since vLLM is a rapidly developing platform, some things may break.

Selecting --exp-b12x without local-build flags or customizations pulls the separately tested eugr/spark-vllm-b12x:latest image and tags it as vllm-node-b12x.

If you want to build only the runner from precompiled vLLM and FlashInfer wheels, specify --use-wheels. This option never falls back to compiling missing wheels: if a wheel cannot be downloaded or found locally, the command stops with an error. To build the latest vLLM from the main branch, use --rebuild-vllm; to target a specific repository, release, or commit, set --vllm-repo and/or --vllm-ref. To build a private or already-available checkout without cloning it inside Docker, use --vllm-source-dir.

Similarly, --rebuild-flashinfer, --flashinfer-ref, and --apply-flashinfer-pr control the FlashInfer build and force the local build path.

Local builds include a CUDA-on-WSL memory-reporting fix in both compiled vLLM wheels and runners built with --use-wheels. On integrated NVIDIA GPUs under WSL, vLLM keeps CUDA's reported free memory instead of replacing it with guest RAM availability. Native Linux UMA accounting and proactive allocator-cache release keep their upstream behavior.

Regular and B12X source builds also trim unused glibc CPU heap pages before the final KV cache sizing measurement, after profiling cleanup. On Spark's shared CPU/GPU memory, this prevents freed loader/profiling buffers retained by glibc from reducing the KV budget. API servers and workers still trim after startup garbage collection and warmup. Both trims are included in exported vLLM wheels; platforms without malloc_trim skip them.

Runtime images also set VLLM_WSL2_ENABLE_PIN_MEMORY=1 by default. Pass -e VLLM_WSL2_ENABLE_PIN_MEMORY=0 to launch-cluster.sh or docker run to opt out.

Regular and B12X source builds include a targeted patch based on vLLM PR #58028. When --api-key is configured, the Python frontend requires authentication for every route except /health, /ping, /load, and /version; CORS preflight requests also remain exempt. This includes protecting /metrics, /tokenize, and the API docs. The patch is included in exported vLLM wheels and leaves the Rust frontend unchanged.

B12X source builds fix the shared NVFP4/MXFP4 scale-swizzle accessor for FlashInfer's split intrinsics.reference module, while retaining support for older FlashInfer refs. The fix is applied before compilation and included in exported vLLM wheels.

QUICK START (USING RECIPES)

Recipe launches check that the model declared in the recipe is in the local Hugging Face cache. If it is missing, the runner stops and asks you to rerun the same command with --setup. Dry runs do not require downloaded models.

Single Spark

Check out locally. Do it on the head node of the cluster. This will build the image, download the model and launch it in the container.

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker
./run-recipe.sh recipes/qwen3.8-flash-next-nvfp4-solo.yaml --solo --setup

Dual Sparks (or more)

Before you start, make sure you connect your Sparks together and enable passwordless SSH as described in our Networking Guide. You can also check out NVIDIA's Connect Two Sparks Playbook, but using our guide is the best way to get started. The guide includes instructions for 3-node Spark mesh clusters.

The experimental cluster setup helper can configure CX7 networking and mutual passwordless SSH from the head node using the nodes' management IP addresses. It supports direct connections, switches, and closed rings, saves the head's launch .env after verification, and includes dry-run and restore modes. Use --mtu BYTES to override the default CX7 MTU of 9000. It also adds Docker group membership on every node; --doctor checks and repairs an existing saved setup.

Check out locally. Do it on the head node of the cluster. This will build the image, download and distribute the model and launch the cluster.

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker
./run-recipe.sh recipes/deepseek-v4-flash-vision-exp.yaml --setup

QUICK START (USING LAUNCHER)

These examples show image preparation, model download, and launch-cluster.sh commands separately. Qwen3.8-27B uses the regular image (vllm-node). The Qwen3.8 Flash Next and DeepSeek Vision examples match the shortcut above and use the B12X image (vllm-node-b12x). Choose the example for your setup.

Check out the repository

Run the commands on your Spark, or on the head node of your cluster:

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker

Single Spark: Qwen3.8-27B NVFP4 (regular image)

This matches the Qwen3.8-27B NVFP4 with DFlash2 recipe in solo mode. It uses FP8 KV cache, InstantTensor loading, and the DFlash2 draft model for speculative decoding, with a maximum context length of 262144 tokens.

Pull the tested regular image and download both the main and draft models:

./build-and-copy.sh
./hf-download.sh nvidia/Qwen3.8-27B-NVFP4
./hf-download.sh z-lab/Qwen3.8-27B-DFlash2

Launch the server:

./launch-cluster.sh --solo -t vllm-node \
  exec vllm serve nvidia/Qwen3.8-27B-NVFP4 \
    --host 0.0.0.0 \
    --port 8000 \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.8 \
    --max-model-len 262144 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 16384 \
    --enable-chunked-prefill \
    --async-scheduling \
    --enable-prefix-caching \
    --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.8-27B-DFlash2","num_speculative_tokens":8,"draft_tensor_parallel_size":1}' \
    --load-format instanttensor \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml \
    --enable-auto-tool-choice \
    --tensor-parallel-size 1

Single Spark: Qwen3.8 Flash Next FP4 Local

This matches the solo Qwen3.8 Flash Next recipe, using nvidia/Qwen3.8-Flash-Next-FP4-Local, with PLE tables offloaded to disk through io_uring and speculative decoding. It uses the B12X model loader and a maximum context length of 262144 tokens.

Pull the tested B12X image and download the model:

./build-and-copy.sh --exp-b12x
./hf-download.sh nvidia/Qwen3.8-Flash-Next-FP4-Local

Launch the server:

./launch-cluster.sh --solo -t vllm-node-b12x \
  -e PYTORCH_ALLOC_CONF=per_process_memory_fraction:0.8 \
  -e VLLM_PLE_TABLE_MEMORY=disk \
  -e B12X_DISK_BACKEND=io_uring \
  -e CUTE_DSL_ARCH=sm_121a \
  -e SAFETENSORS_FAST_GPU=1 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e VLLM_SSM_CONV_STATE_LAYOUT=DS \
  -e VLLM_USE_AOT_COMPILE=1 \
  -e VLLM_USE_MEGA_AOT_ARTIFACT=1 \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e B12X_POLICY_MODE=auto \
  -e VLLM_MXFP8_LM_HEAD=1 \
  -e VLLM_B12X_MOE_FP4_LAYER_MAX_INPUT_SCALE=w13 \
  -e VLLM_B12X_MOE_FP4_FORCE_A16=1 \
  -e B12X_W4A16_A4_PREFILL=1 \
  exec vllm serve nvidia/Qwen3.8-Flash-Next-FP4-Local \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --pipeline-parallel-size 1 \
    --mamba-cache-mode align \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --dtype bfloat16 \
    --kv-cache-dtype fp8 \
    --quantization modelopt_mixed \
    --block-size 16 \
    --load-format b12x \
    --max-model-len 262144 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 4096 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
    --gdn-decode-kernel b12x \
    --linear-backend b12x \
    --moe-backend b12x \
    --no-enable-flashinfer-autotune \
    --mm-encoder-tp-mode data \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_xml \
    --enable-auto-tool-choice \
    --compilation-config '{"pass_config":{"fuse_act_quant":true}}' \
    --gpu-memory-utilization 0.8

Launcher script uses host networking by default. To use Docker po