SELF-HOSTING • TUTORIAL

Running a 35B Mixture-of-Experts LLM on an 8 GB GPU with llama.cpp

How a Qwen3.6-35B-A3B MoE model runs comfortably on a mid-range Turing GPU — 33 tok/s decode, 150 tok/s prompt processing, 128K context, 7 GB VRAM. No cloud, no API keys, fully private.

TL;DR: Keep the MoE experts on the CPU and only the dense trunk on the GPU, add a quantized KV cache, MTP speculative decoding, and a few llama.cpp flags that are easy to get wrong. Full working recipe below.

What you need

Nothing exotic — a mid-range desktop from a few years ago is enough. This is the exact setup this guide was written against:

The model

Mixture-of-Experts (MoE) models activate only a fraction of their parameters per token. The Qwen3.6-35B-A3B has 35B total parameters but only ~3B active — that is what makes it feasible on consumer hardware. It also ships with a built-in MTP head (Multi-Token Prediction), which llama.cpp can use for speculative decoding.

Download the Q4_K_M GGUF (≈ 22.7 GB) — the sweet spot between size and quality:

wget -P ~/llamacpp/models https://huggingface.co/example-org/Qwen3.6-35B-A3B-UD-GGUF/resolve/main/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

Build llama.cpp with CUDA

Use the v0.4.1 release. Two things trip people up on Ubuntu 26.04: the system GCC is too new for CUDA (use GCC 12), and you need the CUDA dev libraries, not just the driver.

# 1. CUDA 13.3 from the NVIDIA repo (Ubuntu's own 13.1 breaks with glibc 2.43)
sudo apt install cuda-libraries-dev-13-3 cuda-nvcc-13-3 cuda-minimal-build-13-3

# 2. GCC 12 (GCC 15 is too new for the CUDA toolchain)
sudo apt install gcc-12 g++-12

# 3. Clone and build
git clone --depth 1 --branch v0.4.1 https://github.com/ggml-org/llama.cpp
cmake -B llama.cpp/build-cuda -G Ninja \
  -DGGML_CUDA=ON -DLLAMA_CURL=ON \
  -DGGML_CUDA_FA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=75 \
  -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-12 \
  -DCMAKE_C_COMPILER=/usr/bin/gcc-12 \
  -DCMAKE_CXX_COMPILER=/usr/bin/g++-12
ninja -C llama.cpp/build-cuda -j 16

Note: -DCMAKE_CUDA_ARCHITECTURES=75 is for Turing (RTX 20-series). For Ampere use 86, for Ada 89.

The start script — every flag explained

This is the production start script (used via systemd, see below). The result of a lot of trial and error — each flag earns its place:

#!/bin/bash
# llama-server start script (systemd)
set -e
MODEL="/home/user/llamacpp/models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf"
BIN="/home/user/llamacpp/llama.cpp/build-cuda/bin/llama-server"
LOG="/home/user/llamacpp/server.log"
API_KEY="$(cat /home/user/llamacpp/api_key.txt)"

exec "$BIN" \
  -m "$MODEL" \
  --n-cpu-moe 41 \
  --load-mode none \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --spec-type draft-mtp -n 3 \
  --no-reasoning-preserve \
  --n-predict 4096 \
  --kv-unified-per-slot 131072 \
  --parallel 2 \
  --threads 16 \
  --threads-batch 16 \
  --host 0.0.0.0 \
  --port 8090 \
  --api-key "$API_KEY" \
  >> "$LOG" 2>&1

Run it as a systemd service

A 22.7 GB model takes a while to load — systemd with a generous timeout and auto-restart is the right way to run it. The 30-second delay after boot gives the system time to settle:

# /etc/systemd/system/llama-server.service
[Unit]
Description=Local LLM Server (llama.cpp, Qwen3.6-35B-A3B MoE)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=user
Group=user
WorkingDirectory=/home/user/llamacpp
ExecStartPre=/bin/sleep 30
ExecStart=/home/user/llamacpp/llama-server-start.sh
Restart=on-failure
RestartSec=10
TimeoutStartSec=600
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable llama-server   # autostart on boot
sudo systemctl start llama-server
systemctl is-active llama-server     # → active

Security: the server binds 0.0.0.0 — generate a key with openssl rand -hex 24 > api_key.txt, chmod 600 it, read it from the start script. Never hardcode it in the unit file (world-readable).

Verify it's working

# Health check (no auth needed)
curl http://server01.local:8090/health

# Chat completion
curl http://server01.local:8090/v1/chat/completions \
  -H "Authorization: Bearer $(cat ~/llamacpp/api_key.txt)" \
  -H "Content-Type: application/json" \
  -d '{"model":"local","messages":[{"role":"user","content":"Say hello in one sentence."}]}'

If the reply comes back empty, check server.log for finish: length — the reasoning block ate the output budget and --n-predict 4096 is missing.

Measured results

Real numbers from the server log (slot print_timing), Qwen3.6-35B-A3B Q4_K_M, 128K context, MTP speculative decoding on:

Pitfalls we hit, so you don't have to

1. CUDA 13.1 from Ubuntu repos breaks with glibc 2.43 — compile errors in mathcalls.h (rsqrt conflict). Use CUDA 13.3 from the NVIDIA repo — fixed natively.
2. GCC 15 is too new for CUDA — install gcc-12/g++-12 and point CMake at them via -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-12.
3. MoE + speculative decoding: draft cap 2–3, not 8 — with MoE, the expert union grows with draft length — longer drafts are slower than no drafts at all.
4. Empty replies with reasoning models — Qwen3.6's thinking block counts against the output budget. Symptom: finish: length with empty content. Fix: --n-predict 4096 + --no-reasoning-preserve.
5. Don't trust the context length on the model card — 262K/1M are lab numbers. With KV quantization, ~128K is the realistic ceiling — still plenty.
6. Port conflicts on a busy host — check ss -tlnp and docker ps before binding — pick a free port (8090 worked here).

Bottom line: a 35B MoE model on an 8 GB GPU is not a hack — it's a sensible architecture choice. 33 tok/s is entirely usable for chat, agents, and tool-calling, and everything runs on your own hardware.