moonshotai/Kimi-K2.6

Overview

Kimi K2.6 is an open-source, native multimodal agentic model built through continual pretraining on approximately 15 trillion mixed visual and text tokens atop Kimi-K2-Base. It seamlessly integrates vision and language understanding with advanced agentic capabilities, instant and thinking modes, as well as conversational and agentic paradigms.

Prerequisites

• vLLM version: >= 0.25.0 (required for the B300 EAGLE3 and native CPU KV offload paths below)
• Hardware (INT4): 8× H200 GPUs (verified), or equivalent aggregate VRAM (~640 GB)
• Hardware (B300 / NVFP4): 4× B300 (verified); use vllm/vllm-openai:latest
• AMD support: 8x MI300X / MI325X / MI355X with ROCm 7.2.1 and Python 3.12

NVIDIA B300: NVFP4 with Eagle3

export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

vllm serve nvidia/Kimi-K2.6-NVFP4 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 64 \
  --gpu-memory-utilization 0.90 \
  --attention-backend TOKENSPEED_MLA \
  --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --max-cudagraph-capture-size 2048 \
  --speculative-config '{"method":"eagle3","model":"lightseekorg/kimi-k2.6-eagle3-mla","num_speculative_tokens":4}'

Native CPU KV offload

Select Simple in the command builder's KV Offload row to extend the prefix cache into host DRAM with SimpleCPUOffloadConnector. The shared option uses 220 GiB per rank by default. The verified B300 TP4 run used 1,199 GiB total (299.75 GiB per rank); the equivalent explicit command is:

CPU_OFFLOAD_BYTES_PER_RANK=321854111744

vllm serve nvidia/Kimi-K2.6-NVFP4 \
  --tensor-parallel-size 4 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 64 \
  --gpu-memory-utilization 0.90 \
  --attention-backend TOKENSPEED_MLA \
  --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' \
  --disable-hybrid-kv-cache-manager \
  --kv-transfer-config "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use_per_rank\":${CPU_OFFLOAD_BYTES_PER_RANK},\"lazy_offload\":false}}"

Decode context parallelism

For higher concurrency on the B300 NVFP4 path, add --decode-context-parallel-size 4 (match TP size). The command builder's Advanced row also exposes DCP for this recipe.

AMD MI300X/MI325X

On 8x MI300X or MI325X (gfx942), use the standard W4A16 MoE path with AITER and INT4 QuickReduce.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.6 \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --tool-call-parser kimi_k2 \
  --enable-auto-tool-choice \
  --reasoning-parser kimi_k2 \
  --mm-encoder-tp-mode data

AMD MI350X/MI355X

On 8x MI350X or MI355X (gfx950), add --moe-backend flydsl to use the optimized FlyDSL W4A16 MoE kernel. Keep LoRA disabled for this path.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

vllm serve moonshotai/Kimi-K2.6 \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --mm-encoder-tp-mode data \
  --moe-backend flydsl \
  --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'

Notes:

• The FlyDSL INT4 MoE path does not support expert parallelism; do not add --enable-expert-parallel.
• Keep --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}'; it is required for this FlyDSL path on MI350X / MI355X.
• vLLM has tuned MI350X/MI355X FlyDSL configs for this Kimi shape at TP=8 and TP=4.
• Keep vLLM's default block size unless you are tuning long-context throughput; --block-size 64 is safe to try.

Client Usage

Once the vLLM server is running, consume it via the OpenAI-compatible API:

import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="moonshotai/Kimi-K2.6",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")

Troubleshooting

• OOM errors: Lower --gpu-memory-utilization or adjust TP/EP to match your GPU count.
• Vision encoder performance: Use --mm-encoder-tp-mode data to run the vision encoder in data-parallel mode. The encoder is small, so TP adds communication overhead with little gain.
• Unique multimodal inputs: Pass --mm-processor-cache-gb 0 to avoid caching overhead. For repeated inputs, --mm-processor-cache-type shm uses host shared memory for better performance at high TP settings.
• MoE kernel tuning: Use the benchmark_moe script from vLLM to tune Triton kernels for your specific hardware.
• Async scheduling: Enabled by default for better throughput. Disable if you encounter issues, and file a bug report to vLLM.

References

• Kimi-K2.6 on Hugging Face
• NVIDIA Kimi-K2.6-NVFP4 on Hugging Face
• vLLM multimodal inputs guide
• vLLM Expert Parallelism docs
• vLLM NixlConnector usage guide