When you need this page
If your Provider node is going to use SGLang to handle inference tasks, follow this page first to start the SGLang HTTP service. SGLang provides an OpenAI-compatible interface, so on the Provider Agent side you need to use --api-format sglang.
The model names in the Token supply examples, such as qwen3.6:35b and gemma4:26b, are the platform model codes used when starting the Provider Agent. SGLang loads a Hugging Face model ID or a local model directory; the "model name" shown in the console is used to download and start the model, while the "model ID" is used for the OpenAI-compatible model identifier that SGLang exposes and for the Provider Agent's --runtime-model-id.
Supported systems and characteristics
SGLang's standard getting-started path targets mainly Linux, Python 3.10+, and NVIDIA CUDA GPUs. The official materials also provide pages for AMD GPU, Intel Xeon CPU, Apple Silicon with Metal, TPU, Ascend NPU, Intel XPU, and other hardware, but the capabilities and installation methods of the different backends vary considerably.
Main advantages:
- Built for production inference, with an emphasis on low latency and high throughput
- Supports OpenAI-compatible endpoints such as
/v1/chat/completions,/v1/completions, embedding, and vision - Supports RadixAttention, prefix caching, continuous batching, chunked prefill, speculative decoding, and various parallelism strategies
Main limitations:
- Installation depends on a combination of CUDA, ROCm, PyTorch, FlashInfer, and other versions, making the environment more complex than Ollama
- The available features differ across hardware backends
Installing SGLang
A general Linux/NVIDIA environment can install using uv:
pip install --upgrade pippip install uvuv pip install "sglang[all]"If you use Docker, you can run the official image directly. The example below mounts the Hugging Face cache into the container and exposes the OpenAI-compatible API on port 30000:
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.6-35B-A3B \ --context-length 262144 \ --host 0.0.0.0 \ --port 30000If the model requires Hugging Face authentication, replace <secret> with your Hugging Face token.
Preparing the model name
SGLang's --model-path accepts a Hugging Face model ID or a local model directory; it does not download models by Ollama's model:tag directly.
In practice, first open the Provider API key page, select the target model and the SGLang inference framework in the Token supply examples, then click "Copy model name" and "Copy model ID" separately. The model name is used as the download source for --model-path; the model ID is the model used in /v1/models and verification requests, and it is also the Provider Agent's --runtime-model-id / PROVIDER_AGENT_RUNTIME_MODEL_ID.
Common mappings can be handled as follows; the authoritative values are those shown as "model name / model ID" on the page:
| Token supply example model name | SGLang model name | SGLang model ID | Notes |
|---|---|---|---|
qwen3.6:35b | Qwen/Qwen3.6-35B-A3B | Qwen/Qwen3.6-35B-A3B | Good as a long-context example |
gemma4:26b | google/gemma-4-26B-A4B-it | google/gemma-4-26B-A4B-it | The specific checkpoint follows the actual download and platform configuration |
By default SGLang uses the value of --model-path as the server-side model name. Generally, keeping this name is fine:
python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.6-35B-A3B \ --context-length 262144 \ --host 0.0.0.0 \ --port 30000If you really need to use --served-model-name to override the returned name, it should also be set to the SGLang model ID in the table above, for example Qwen/Qwen3.6-35B-A3B, not an Ollama-style qwen3.6:35b. The Provider Agent's --runtime-model-id must match this server-side model ID.
Setting the context window
The corresponding context parameter in SGLang is --context-length. This value should be set according to the "context window" size required by the target model, and should not default to the same number for all models. The 262144 in the example below applies only to models that explicitly support a 256K context window; if the model requires 128K, set it to 131072.
python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.6-35B-A3B \ --served-model-name Qwen/Qwen3.6-35B-A3B \ --context-length 262144 \ --tp-size 8 \ --mem-fraction-static 0.8 \ --reasoning-parser qwen3 \ --host 0.0.0.0 \ --port 30000Parameter notes:
--context-length 262144indicates in the example that the model's maximum context window is 256K tokens; the actual value should be filled in according to the target model's requirement--tp-sizeshould be adjusted according to the number of GPUs and the model size; it is not a fixed value--mem-fraction-staticcan be lowered when VRAM is tight--reasoning-parser qwen3applies only to models that need Qwen3 reasoning parsing
Notes:
- The model itself must support the target context window size you set
max_tokensonly controls the maximum number of tokens generated in a single request; it does not expand the context window
Verifying the SGLang service
View the model list:
curl http://127.0.0.1:30000/v1/modelsDo a single non-streaming inference:
curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ --data-raw '{ "model": "Qwen/Qwen3.6-35B-A3B", "messages": [ { "role": "user", "content": "Reply in one sentence: the SGLang service is available." } ], "max_tokens": 64, "temperature": 0 }'Then check streaming output:
curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ --data-raw '{ "model": "Qwen/Qwen3.6-35B-A3B", "messages": [ { "role": "user", "content": "List three health check points for a local inference service." } ], "stream": true, "max_tokens": 128 }'If you configured --api-key skypool-test when starting SGLang, all requests must add:
-H "Authorization: Bearer skypool-test"Starting the Provider Agent
Once the SGLang service is available, run the preflight first:
./token-provider-agent preflight start \ --model qwen3.6:35b \ --base-url http://127.0.0.1:30000 \ --api-format sglang \ --runtime-model-id Qwen/Qwen3.6-35B-A3BAfter the preflight passes, start it for real:
./token-provider-agent start \ --api-key stp-... \ --model qwen3.6:35b \ --base-url http://127.0.0.1:30000 \ --api-format sglang \ --runtime-model-id Qwen/Qwen3.6-35B-A3BIf SGLang has an inference framework API Key enabled, also pass:
--runtime-api-key skypool-test