Z.ai/GLM-4.5-Air
A 106B MoE served from a GGUF checkpoint via llama.cpp on one A100.
View on Hugging FaceA 106B MoE served from an Unsloth GGUF checkpoint via llama.cpp instead of
vLLM, on one A100 40 GB. Because Modelplane treats llama.cpp like any other
OpenAI-compatible container, the only changes from a vLLM deployment are the
image and args: the container is still named engine and listens on :8000.
vLLM can’t load this Unsloth quantization format. llama.cpp can, and -hf
pulls the checkpoint straight from Hugging Face at startup, so a one-time
deployment needs no ModelCache.
The model is bigger than one A100’s VRAM, so --n-cpu-moe offloads the MoE
expert tensors to host RAM and the GPU runs the active path and KV cache.
That’s how a 106B model fits one A100 instead of a multi-GPU node. Apply the
platform side first, then the ML side.
Validated deployments
Platform
# A single A100 40GB on GKE. GLM-4.5-Air is a ~106B MoE; a 4-bit GGUF doesn't
# fit one A100's VRAM, but llama.cpp offloads the expert tensors to host RAM
# (see the ModelDeployment's --n-cpu-moe), so the GPU only holds the active path
# + KV cache. One A100, not a multi-GPU node.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
name: gke-a100-40-1x
spec:
description: "GKE a2-highgpu-1g, 1x NVIDIA A100 40GB"
provisioning:
provider: GKE
gke:
machineType: a2-highgpu-1g
diskSizeGb: 200
accelerator:
type: nvidia-tesla-a100
count: 1
devices:
- name: gpu
claim: DRA
driver: gpu.nvidia.com
deviceClassName: gpu.nvidia.com
count: 1
attributes:
architecture: { string: Ampere }
capacity:
memory: { value: "40960Mi" }
# A GKE cluster with a single A100 node offering the class above.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
name: gke-a100
labels:
modelplane.ai/region: us-west
spec:
cluster:
source: GKE
gke:
region: us-west1
nodePools:
- name: gpu-a100
className: gke-a100-40-1x
nodeCount: 1
minNodeCount: 1
maxNodeCount: 1
zones:
- us-west1-b
Deployment
# GLM-4.5-Air (~106B MoE) served from an Unsloth GGUF via llama.cpp instead of
# vLLM, on a SINGLE A100. Modelplane treats the engine as any OpenAI-compatible
# container, so the only changes from a vLLM deployment are the image and args:
# the container is still named `engine` and listens on :8000. vLLM can't load
# Unsloth's UD- dynamic quants; llama.cpp can, and `-hf` pulls the quant from
# HuggingFace at startup (no ModelCache needed for a one-off).
#
# The model is bigger than one A100's VRAM, so --n-cpu-moe offloads the MoE
# expert tensors to host RAM; the GPU runs the active path. That's how a 106B
# model fits one A100 instead of a multi-GPU node. --port 8000 because llama.cpp
# defaults to 8080 and Modelplane scrapes 8000.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
name: glm-air
namespace: ml-team
spec:
replicas: 1
template:
spec:
engines:
- name: glm
members:
- role: Standalone
nodeSelector:
devices:
- name: gpu
count: 1
selectors:
- cel: |
device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("35Gi")) >= 0
template:
spec:
containers:
- name: engine
image: ghcr.io/ggml-org/llama.cpp:server-cuda
args:
- "-hf"
- "unsloth/GLM-4.5-Air-GGUF:IQ4_XS"
# llama.cpp's equivalent of --served-model-name.
- "-a"
- "$(MODELPLANE_SERVED_MODEL_NAME)"
- "--host"
- "0.0.0.0"
- "--port"
- "8000"
- "-ngl"
- "999"
- "--n-cpu-moe"
- "99"
- "--jinja"
- "-c"
- "8192"
# Exposes the deployment as one model a caller can name.
# Name the model, ml-team/glm-air, in a request to an InferenceGateway
# that serves it.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
name: glm-air
namespace: ml-team
spec:
endpoints:
- name: glm-air
selector:
matchLabels:
modelplane.ai/deployment: glm-air