vLLM Local Model Setup

When you need this page

If your Provider node is going to use vLLM to serve inference tasks, follow this page first to start the vLLM service and model instance. vLLM exposes an OpenAI-compatible interface, which is different from Ollama's /api interface; on the Provider Agent side you need to use --api-format vllm.

The model names in the Token supply examples, such as qwen3.6:35b and gemma4:26b, are the platform model codes used when starting the Provider Agent. vLLM itself loads a Hugging Face model ID or a local model directory; the "model name" shown in the console is used to download and start the model, while the "model ID" is used for the OpenAI-compatible model identifier that vLLM exposes and for the Provider Agent's --runtime-model-id.

Supported systems and characteristics

vLLM is primarily aimed at Linux servers and GPU inference. The standard GPU path in the official documentation focuses on Linux, Python 3.10 to 3.13, and NVIDIA CUDA or AMD ROCm.

Main advantages:

• Strong throughput and concurrency, suitable for long-running online Provider nodes
• Supports OpenAI-compatible interfaces such as /v1/models and /v1/chat/completions
• Supports PagedAttention, continuous batching, prefix caching, chunked prefill, and multi-GPU parallelism
• Supports Hugging Face model IDs, local model directories, and various quantization formats

Main limitations:

• Installation and driver environment are more complex than Ollama; you need to pay attention to CUDA, ROCm, PyTorch, GPU memory, and parallelism
• A single vllm serve typically serves one primary model; serving multiple models generally requires multiple processes or additional routing
• --max-model-len cannot make a model exceed the context limit that its own training or configuration supports

Installing vLLM

For an NVIDIA CUDA environment, using uv to create an isolated environment is recommended:

uv venv --python 3.12 --seedsource .venv/bin/activateuv pip install vllm --torch-backend=auto

An AMD ROCm environment needs to use the ROCm wheel source; refer to the official vLLM installation page for the exact version:

uv venv --python 3.12 --seedsource .venv/bin/activateuv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/

You can also use the official OpenAI-compatible Docker image directly:

docker run --runtime nvidia --gpus all \  -v ~/.cache/huggingface:/root/.cache/huggingface \  --env "HF_TOKEN=$HF_TOKEN" \  -p 8000:8000 \  --ipc=host \  vllm/vllm-openai:latest \  --model Qwen/Qwen3.6-35B-A3B \  --max-model-len 262144

If the model requires Hugging Face authentication, set HF_TOKEN first. On the first launch, vLLM will download the model to the Hugging Face cache directory.

Preparing the model name

vLLM's --model accepts a Hugging Face model ID or a local path; it does not accept an Ollama-style name:tag as a download target.

In practice, first open the Provider API Keys page, select the target model and the vLLM inference framework in the Token supply examples, then click "Copy model name" and "Copy model ID" respectively. The model name is used as the download source for vllm serve; the model ID is the model in /v1/models and in verification requests, and is also the Provider Agent's --runtime-model-id / PROVIDER_AGENT_RUNTIME_MODEL_ID.

The common correspondence can be understood as follows; ultimately, use the values shown as "model name / model ID" on the page:

By default, vLLM uses the value of --model as the server-side model name. Generally, keeping this name is fine:

vllm serve Qwen/Qwen3.6-35B-A3B \  --max-model-len 262144 \  --host 0.0.0.0 \  --port 8000

If you really need to use --served-model-name to override the returned name, it should also be set to the vLLM model ID from the table above, for example Qwen/Qwen3.6-35B-A3B; do not set it to the Ollama-style qwen3.6:35b. The Provider Agent's --runtime-model-id must match this server-side model ID.

Setting the context window

The vLLM field that corresponds to the context window is max_model_len, and the command-line argument is --max-model-len. This value should be set according to the "context window" size that the target model requires; you cannot default all models to the same number. The 262144 in the examples below only applies to models that explicitly support a 256K context window; if the model requires 128K, it should be set to 131072.

vllm serve Qwen/Qwen3.6-35B-A3B \  --served-model-name Qwen/Qwen3.6-35B-A3B \  --max-model-len 262144 \  --dtype auto \  --host 0.0.0.0 \  --port 8000 \  --api-key runtime-...

When you need multiple GPUs, increase tensor parallel according to your machine:

vllm serve Qwen/Qwen3.6-35B-A3B \  --served-model-name Qwen/Qwen3.6-35B-A3B \  --max-model-len 262144 \  --tensor-parallel-size 4 \  --host 0.0.0.0 \  --port 8000

Notes:

• --max-model-len is the combined context length of the prompt and the output
• max_tokens in the OpenAI request body is only the maximum number of tokens to generate this time, not the context window
• If the model configuration itself does not support the target context window size, forcing the setting may cause errors, degraded quality, or GPU memory overflow
• If you hit OOM at startup, first reduce --max-model-len, reduce concurrency, adjust quantization, or increase GPU parallelism

Verifying the vLLM service

If you configured --api-key runtime-... at startup, you must also include the Authorization header when verifying.

View the model list:

curl http://127.0.0.1:8000/v1/models \  -H "Authorization: Bearer runtime-..."

Run a non-streaming inference:

curl http://127.0.0.1:8000/v1/chat/completions \  -H "Authorization: Bearer runtime-..." \  -H "Content-Type: application/json" \  --data-raw '{    "model": "Qwen/Qwen3.6-35B-A3B",    "messages": [      {        "role": "user",        "content": "Reply in one sentence: the vLLM service is available."      }    ],    "max_tokens": 64,    "temperature": 0  }'

If you did not configure --api-key, just remove the Authorization line.

Starting the Provider Agent

Once the vLLM service is available, start the Provider Agent:

./token-provider-agent preflight start \  --model qwen3.6:35b \  --base-url http://127.0.0.1:8000 \  --api-format vllm \  --runtime-model-id Qwen/Qwen3.6-35B-A3B \  --runtime-api-key runtime-...

After preflight passes, start it for real:

./token-provider-agent start \  --api-key stp-... \  --model qwen3.6:35b \  --base-url http://127.0.0.1:8000 \  --api-format vllm \  --runtime-model-id Qwen/Qwen3.6-35B-A3B \  --runtime-api-key runtime-...

If vLLM does not have an inference-framework API Key enabled, you can omit --runtime-api-key.

Reference links

• vLLM installation documentation
• vLLM OpenAI-compatible Server
• vLLM serve CLI
• vLLM Engine Arguments
• Qwen/Qwen3.6-35B-A3B