llama.cpp Local Model Setup

When you need this page

If your Provider node is going to use llama.cpp to serve inference tasks, follow this page first to start llama-server and a GGUF model. The llama.cpp HTTP Server exposes OpenAI-compatible interfaces such as /v1/models and /v1/chat/completions, and on the Provider Agent side you need to use --api-format llama.cpp.

The model names in the Token supply examples, such as qwen3.5:9b and gemma4:26b, are the platform model codes used when starting the Provider Agent. llama.cpp loads a local .gguf file; the "model name" shown in the console is used to confirm the model source, while the "model ID" is recommended as the llama-server API alias and is passed to the Provider Agent's --runtime-model-id.

Supported systems and characteristics

llama.cpp is a good fit for local Providers who want to run GGUF models directly. It can run on many backends such as CPU, Metal, CUDA, Vulkan, and ROCm; the exact performance depends on the version you compile or download, the quantization format, GPU/system memory, and the context window size.

Main advantages:

• Lightweight runtime, suitable for local machines, desktop workstations, and professional nodes
• Supports GGUF models, quantized models, and multimodal projector files
• llama-server provides an OpenAI-compatible API, convenient for Provider Agent preflight and calls
• The context window can be set explicitly via --ctx-size

Main limitations:

• You need to prepare and match the GGUF files, quantization levels, and multimodal projectors yourself
• A large context window significantly increases KV cache usage and can easily trigger out-of-memory on GPU or system memory
• Build methods and performance vary considerably across hardware backends, so you need to benchmark on your own machine

Installing llama.cpp

It is recommended to first consult the official repository documentation to choose an installation method. A common source-build flow is as follows:

git clone https://github.com/ggml-org/llama.cppcd llama.cppcmake -B buildcmake --build build --config Release -j

After the build completes, llama-server is usually located at:

./build/bin/llama-server

If you use an official or community-provided prebuilt package, confirm that the command name is llama-server and that it can print help normally:

./llama-server --help

Preparing the GGUF model

llama.cpp's -m / --model accepts the path to a local GGUF model file. A multimodal model also needs a matching projector file, specified with -mm / --mmproj.

In practice, first open the Provider API Keys page, select the target model and the llama.cpp inference framework in the Token supply examples, then click "Copy model name" and "Copy model ID" respectively. The model name is used to confirm the source of the GGUF you download; the model ID should be used as the -a / --alias for llama-server, and for the model in later verification requests as well as the Provider Agent's --runtime-model-id / PROVIDER_AGENT_RUNTIME_MODEL_ID.

The recommended approach is as follows; ultimately, use the values shown as "model name / model ID" on the page, the GGUF file you actually download, and the /v1/models return:

If the model is a multimodal model, there is usually also a projector file similar to the following:

mmproj-Qwen3.5-9B.gguf

-m and -mm must point to matching files from the same model family; do not mix GGUF / projector files from different models or different versions.

Starting llama-server

Below is a single-model startup example. -m points to the primary model GGUF, -mm points to the multimodal projector, --ctx-size specifies the context window, and -a specifies the model name exposed externally:

./llama-server \  -m Qwen3.5-9B-Q4_K_M.gguf \  --host 127.0.0.1 \  --port 8080 \  -mm mmproj-Qwen3.5-9B.gguf \  --ctx-size 131072 \  -a "Qwen/Qwen3.5-9B"

If you want to add an inference-framework API Key layer to the local service, you can add:

--api-key runtime-...

In that case, subsequent curl requests and the Provider Agent both need to carry the same token.

Setting the context window

The llama.cpp context parameter is --ctx-size. This value should be set according to the "context window" size that the target model requires; you cannot default all models to the same number. The common conversion is: 128K corresponds to 131072 tokens, and 256K corresponds to 262144 tokens.

Notes:

• --ctx-size is the context window at model load time
• max_tokens in the OpenAI request body is only the maximum number of tokens to generate in a single call, not the context window
• The larger the context window, the higher the KV cache usage; if startup fails or you hit OOM while running, first reduce --ctx-size, reduce concurrency, or switch to a different quantization file
• The model itself must support the target context window size you set

Verifying the llama.cpp service

View the model list:

curl http://127.0.0.1:8080/v1/models

Run a non-streaming inference. Use the model ID exposed by -a for model:

curl http://127.0.0.1:8080/v1/chat/completions \  -H "Content-Type: application/json" \  --data-raw '{    "model": "Qwen/Qwen3.5-9B",    "messages": [      {        "role": "user",        "content": "Reply in one sentence: the llama.cpp service is available."      }    ],    "max_tokens": 64,    "temperature": 0  }'

If you configured --api-key runtime-... at startup, all requests need to add:

-H "Authorization: Bearer runtime-..."

Starting the Provider Agent

Once the llama.cpp service is available, run preflight first:

./token-provider-agent preflight start \  --model qwen3.5:9b \  --base-url http://127.0.0.1:8080 \  --api-format llama.cpp \  --runtime-model-id Qwen/Qwen3.5-9B

After preflight passes, start it for real:

./token-provider-agent start \  --api-key stp-... \  --model qwen3.5:9b \  --base-url http://127.0.0.1:8080 \  --api-format llama.cpp \  --runtime-model-id Qwen/Qwen3.5-9B

If llama.cpp has --api-key enabled, also pass:

--runtime-api-key runtime-...

Reference links

• llama.cpp GitHub
• llama.cpp HTTP Server documentation
• llama.cpp multimodal documentation