Colibri is a pure-C inference engine. It runs GLM-5.2 — a 744-billion-parameter Mixture-of-Experts language model — on a consumer machine with roughly 25 GB of RAM. It has zero dependencies. No BLAS. No Python at runtime. No GPU required, though an optional CUDA backend exists. The engine is a single C file: c/glm.c, approximately 2,400 lines. The project is a one-person effort, written and tested entirely on a 12-core laptop.
The name means hummingbird. Tiny creature. Enormous workload.
The model it runs is GLM-5.2, released by Z.ai under MIT license. 744 billion parameters total. 75 Mixture-of-Experts layers, each with 256 experts. Only about 40 billion parameters activate per token. The challenge: 21,504 routed experts totaling roughly 370 GB at int4 precision. They don't fit in RAM. They barely fit on disk. Colibri makes them work anyway.
How it fits
The insight is separating what must stay resident from what can be streamed. Dense components — attention layers, shared experts, embeddings — total about 17 billion parameters at int4, occupying roughly 9.9 GB of RAM. They stay resident. Routed experts — all 21,504 of them, about 19 MB each at int4 — sit on disk at 370 GB total. They are streamed on demand with a per-layer LRU cache. The operating system's page cache acts as a free second-level cache. An expert that was recently read stays in RAM because the OS hasn't evicted it yet. The OS doesn't know it's caching model weights. It doesn't need to.
The cache is adaptive. At startup, Colibri reads MemAvailable and auto-sizes the expert cache to fit within available RAM, projecting the full working set — KV cache, MTP row, reconstruction buffers — so the OOM killer never fires. The cache learns. The engine records expert usage patterns to .coli_usage. At startup, it pre-pins the hottest experts into spare RAM. A live tier adaptation mode swaps cold experts for hot ones at turn boundaries. The cache gets smarter the longer it runs.
The I/O is async. Expert readahead uses WILLNEED — the kernel reads the next expert block while the CPU computes the current one. An experimental router-lookahead prefetch predicts the next layer's routing from the current layer's post-attention state with 71.6% recall, issuing readahead from a dedicated I/O thread. The disk and the CPU work in parallel. The parallelism is the performance.
The attention
GLM-5.2 uses Multi-head Latent Attention (MLA) with compressed key-value cache. Standard attention stores K and V for every head, for every token. At 64 heads, that's a lot of memory. MLA compresses the KV cache to 576 floats per token instead of 32,768 — a 57× reduction. The compression is lossless relative to full attention. Colibri validates token-exact against the reference implementation.
MLA weight absorption, borrowed from DeepSeek, eliminates per-token key/value reconstruction during decode. The query absorbs the kv_b projection. The reconstruction is skipped. The speedup is significant.
GLM-5.2 also uses DSA sparse attention — a "lightning indexer" that selects a top-2048 causal key set per layer. Colibri auto-extracts the selection weights from the model files. Dense attention is quadratic in sequence length. Sparse attention is linear in the selected set size. The sparsity is the speed.
KV-cache persistence: conversations reopen warm across engine restarts via .coli_kv files. The files are approximately 182 KB per token. They are crash-safe. No re-prefill on restart.
Speculative decoding
Speculative decoding uses a draft model to predict the next several tokens, then verifies them against the full model. If the draft is correct, you get multiple tokens per forward pass. If it's wrong, you recompute. The technique is standard. The implementation is not.
Colibri uses GLM-5.2's own MTP (Multi-Token Prediction) head — layer 78 — as the draft model. The head must be int8, not int4. At int4, acceptance collapses below 4%. At int8, acceptance reaches 39-59%, yielding 2.2-2.8 tokens per forward pass. The draft is lossless under sampling via rejection sampling. No separate draft model. No extra memory for draft weights beyond what the inference engine already loads. The MTP head is a few hundred megabytes. The return is a 2-3× speedup.
Grammar-forced speculative drafts handle constrained outputs — JSON, function calling. When the grammar admits exactly one legal byte, the grammar itself injects a pre-accepted draft. No draft head needed. A wrong grammar cannot change the output. Worst case: rejected drafts, no speedup, no harm.
The quantization
Colibri uses integer-dot kernels. Int8 uses AVX2 maddubs — multiply-add unsigned bytes — achieving approximately 119 GFLOP/s. Int4 uses packed representations with per-row scales and dequant-on-use. The routing between int8, int4, and float32 is decided per shape by measurement. The engine profiles each matrix shape at startup and selects the fastest kernel.
The offline converter takes the FP8 checkpoint — 756 GB — and converts it to int4. It downloads one shard at a time, approximately 5 GB, converts it, and deletes it. The full 756 GB never exists on disk at once. The converter is resumable.
The integer-dot approach was chosen over BLAS for two reasons. First, zero dependencies. Second, expert shapes are small — 19 MB each — and BLAS overhead dominates for small matrices. The custom kernels are faster than calling into OpenBLAS for the shapes that matter. The trade-off was measured. The measurement drove the decision.
The performance
On the developer's laptop — WSL2, 12 cores, 25 GB RAM, NVMe via VHDX at approximately 1 GB/s random reads — Colibri achieves roughly 0.05-0.1 tokens per second cold, improving as the cache warms. Community benchmarks show what happens with more hardware:
| Hardware | Throughput | Notes |
|---|---|---|
| Apple M5 Max, 128 GB, Metal backend | 1.83 tok/s | 66% expert hit rate |
| Ryzen AI 9 HX 370, 128 GB | 0.37 tok/s | MTP acceptance 52% |
| Ryzen 9 9950X, PCIe 5.0 NVMe | 0.28 tok/s | Bottleneck flipped to matmul |
| Ryzen AI Max+ 395, 128 GB, Optane | 0.40 tok/s | 71% hit rate |
The project estimates that with enough RAM to fully cache hot experts plus AVX-512/VNNI kernels, 5-15 tokens per second is achievable — interactive speeds for a 744-billion-parameter model. The bottleneck moves from disk to compute as RAM increases. The movement is the scaling path.
What it includes
The engine is a single C file. The CLI provides chat, serve, plan, doctor, convert, and bench subcommands. The HTTP API is OpenAI-compatible — /v1/chat/completions, /v1/models, SSE streaming, usage counts — with a bounded FIFO admission queue and multi-slot KV contexts. The Web UI is a React/TypeScript client, approximately 390 lines. An optional CUDA backend handles resident tensors and hot-expert VRAM tier with multi-GPU support. Windows 11 native build via MinGW-w64 with POSIX-to-Win32 shims. A quality benchmark harness — MMLU, HellaSwag, ARC — exists but is untested at scale.
The project is Apache 2.0. The model weights are MIT. The entire thing was written by one person on a laptop.
Why this matters
Colibri matters for the same reason the Unix philosophy matters. One tool, doing one thing well. Zero dependencies. The entire inference engine for a 744B model fits in a single C file shorter than this blog post. The complexity is in the model architecture, the quantization scheme, the cache policy, the I/O strategy. The code is not simple. The system is. The system has conceptual integrity — it feels like one mind designed it, because one mind did.
It matters because it proves that frontier models don't require frontier hardware. A 744B model runs on a laptop. Slowly, but it runs. The gap between what the cloud provides and what a consumer machine can do is a function of engineering, not physics. The physics says the model must be read from disk. The engineering says: stream it, cache it, prefetch it, quantize it, compress it. The engineering works.
It matters because it is a one-person project. No team. No funding. No infrastructure beyond a laptop. The output is a competitive inference engine for a model that, six months ago, required a datacenter. The democratization of inference is not a policy objective. It is an engineering achievement. The achievement is public. The code is open. The weights are MIT. Anyone with a laptop and an NVMe drive can run a 744B model. The democratization is complete.
References:
- Colibri, github.com/JustVugg/colibri
- GLM-5.2, Z.ai, MIT License.
- Related posts: Chinese models will win the local-first race, Engineering is art and philosophy, The Unix philosophy
Engineering is the through-line. Every topic on this blog — version control, networking, philosophy, economics, AI — connects to the discipline of designing and building systems that work within constraints. The constraint may be compute, attention, time, or complexity. The method is the same: understand the problem, design a solution, verify it works, iterate. The domain provides the specifics. The method is engineering.
A 744-billion-parameter model running on 25 GB of RAM is not a curiosity. It is a proof. The proof is that the gap between cloud and consumer is a function of engineering, not physics. Engineering can close the gap.