Open-Weight Model Serving: The Memory Bill Behind the Licence

The first question asked about an open-weight model is almost always the licence. It is the wrong first question. Open-weight model serving is constrained by memory long before it is constrained by terms, and for most organisations the memory constraint binds so much harder that the licence never becomes the deciding factor.

The arithmetic is unforgiving and it is not hidden. A model's parameter count and its numeric precision tell you, to within a few per cent, how much high-bandwidth memory you need before it serves a single token. Everything else — throughput, batch size, context length — is negotiable. That number is not.

The quick answer

Multiply parameters by bytes per parameter. A 753-billion-parameter model in BF16 needs roughly 1.5 TB of GPU memory for weights alone; at FP8 it needs about 753 GB. An eight-GPU H100 node holds 640 GB. So the largest open-weight models released in 2026 do not fit on a single conventional node in their native precision, regardless of what their licence permits. Sparse mixture-of-experts architectures reduce the compute per token dramatically but not the memory, because every expert must still be resident.

Key takeaways

• Weights memory ≈ parameters × bytes per parameter. BF16 is 2 bytes; FP8 is 1; FP4 is half.
• MoE models decouple compute from memory. DeepSeek-V4.1-Flash activates 8B parameters during prefill and 16B during decode, but all 552B must be held.
• Active-parameter counts in model names describe your compute bill, not your hardware requirement — a distinction marketing rarely clarifies.
• Million-token context windows create a second, variable memory cost that scales with concurrent users, which is why FP4 KV caching is appearing in frontier releases.
• Licences genuinely differ: Apache 2.0 and MIT sit alongside bespoke model licences with their own conditions.
• Open weights are modifiable weights — including the alignment. A 2026 attack reduced refusal behaviour on a 320B model by 77.6%.

The arithmetic that decides everything

A parameter is a number, and numbers occupy space according to their format. BF16 uses two bytes. FP8 uses one. FP4 uses half a byte. Multiply by the parameter count and you have the floor for weights.

That floor is not the total. You also need memory for activations, for the KV cache that holds the attention state of every in-flight request, and for the runtime itself. A practical planning figure adds meaningful headroom on top of the weights — but the weights are the part that cannot be traded away by reducing batch size or serving fewer users.

What makes this concrete is that the current generation of open-weight models publishes its numbers plainly.

The memory columns are arithmetic on the published parameter counts, not vendor figures, and they exclude KV cache and activation overhead. They are floors.

Set them against hardware. Eight H100s at 80 GB give you 640 GB. That is enough for a 320B model at BF16 with nothing left over, or a 552B model at FP8 with room to work. An NVIDIA GB200 NVL72 rack carries 13.4 TB of HBM3E across 72 GPUs, which holds any of these comfortably — and costs accordingly.

Sparse MoE decouples compute from memory

This is the single most misunderstood property of current open-weight models, and the naming convention actively encourages the misunderstanding.

K2-Horizon-375B-A23B describes itself as "a sparse Mixture-of-Experts model that stores 375B parameters and runs 23B per token, with a 512K context window". Read that sentence carefully: stores 375B, runs 23B. The 23B figure tells you the per-token compute. The 375B figure tells you the hardware you must buy.

DeepSeek-V4.1-Flash makes the split sharper still, activating "8B parameters per token during prefill" and "16B during decode" against a 552B backbone. That is a remarkable efficiency achievement — roughly 3% of the model does the work for any given token — and it changes nothing about the fact that the other 97% has to be somewhere the GPU can reach in nanoseconds.

Active parameters are your electricity bill. Total parameters are your capital expenditure. Conflating them is how self-hosting projects get approved on numbers that were never going to work.

The reason all experts must be resident is routing: which experts a token needs is decided at inference time, per token, per layer. There is no reliable way to predict the working set in advance, and paging experts from host memory across a PCIe boundary costs orders of magnitude more latency than the computation saves. UniverseBlend's explainer on the memory limits of AI chips covers why bandwidth rather than raw compute tends to be the binding constraint in serving.

Nothing this size fits on one GPU

A consequence that follows immediately from the table, and that planning documents routinely skip: none of these models runs on a single accelerator. Every one of them has to be split across GPUs, which turns a memory problem into an interconnect problem.

There are two ways to split, and they fail differently. Tensor parallelism divides individual layers across GPUs, so every token requires the devices to exchange partial results many times per forward pass — fast, but only if the link between them is very fast. Pipeline parallelism assigns whole layers to different GPUs, communicating less but leaving devices idle while they wait their turn. Real deployments combine both, and expert parallelism adds a third axis for MoE models, distributing experts across devices so that routing becomes a network operation.

This is why rack-scale systems exist as products rather than as collections of servers. NVIDIA's GB200 NVL72 links 72 Blackwell GPUs with 130 TB/s of NVLink bandwidth, which is the specification that matters for a model spread across all of them — the memory is only usable as one pool if the fabric can move partial results at something approaching memory speed.

The practical implication for anyone costing this: two nodes with 640 GB each are not equivalent to one node with 1.28 TB. Whether they behave as a single pool depends entirely on what connects them, and a model sharded across a slow link can be dramatically slower than a smaller model that fits in one place. Capacity and topology have to be evaluated together.

Context length is a second, variable bill

Weights are fixed. The KV cache is not.

Every in-flight request holds attention state proportional to its context length, and that state lives in the same GPU memory as the weights. A model advertising a million-token context window is advertising a capability whose memory cost scales with how many users exercise it simultaneously. One user at a million tokens and a thousand users at a thousand tokens are very different provisioning problems that produce superficially similar aggregate numbers.

Frontier releases have started attacking this directly. DeepSeek-V4.1-Flash uses FP4 KV caching in E2M1 format — quantising not the weights but the attention state, which is where the variable cost lives. K2-Horizon ships a 512K native context; GLM-5.3 and DeepSeek-V4.1-Flash both reach a million tokens.

The practical planning consequence: size your deployment against realistic concurrent context, not against the advertised maximum. A serving configuration that works beautifully in a demo with one long document can fail under twenty concurrent users at a quarter of the length.

Quantisation, and what you actually give up

Quantisation is how most organisations make the arithmetic work, and the trade is more favourable than it was two years ago.

FP8 has become close to a default: GLM-5.3 ships tensors in "BF16 · F8_E4M3 · F32", and GLM-5.3-Flash ships in block-FP8 natively rather than as a post-hoc conversion. When a model is trained or released with a quantised format as a first-class artefact, the quality question is substantially settled by the people who built it. That is meaningfully different from quantising a BF16 checkpoint yourself and hoping.

What you give up is harder to state in one number than vendors imply, because the loss is uneven. Aggregate benchmark scores tend to move little. Behaviour at the tails — long-context recall, rare-token handling, arithmetic, structured output adherence — moves more, and moves in ways an average does not reveal. If you quantise, evaluate on your own workload rather than on published scores; UniverseBlend makes the general case for building your own evaluation instead of trusting benchmark numbers, and quantisation is where that advice pays fastest.

The licence still matters — just second

Having established that memory decides feasibility, the licence decides what you may do with a deployment that is already feasible. The current field genuinely varies.

DeepSeek-V4.1-Flash is released under the MIT licence and K2-Horizon-375B-A23B under Apache 2.0 — both permissive, both familiar to any legal team, both imposing essentially no use restrictions. GLM-5.3 ships under a bespoke "glm-5.3" licence, which like most model-specific licences needs actual reading rather than pattern-matching to an open-source template.

Three questions are worth asking of any model licence before it reaches procurement: whether commercial use is permitted without a revenue threshold; whether outputs may be used to train other models; and whether attribution or use-policy conditions attach downstream. Permissive licences answer all three trivially. Bespoke ones frequently do not, and the differences are the kind that surface late.

There is also a regulatory dimension that is newer than most licence reviews account for. Under the EU AI Act's deployer obligations, the documentation a model provider supplies is the raw material for your own compliance file — and if you self-host, some of what a provider would have supplied becomes yours to produce.

The safety question nobody costs

Open weights are modifiable weights, and that includes the alignment.

A September 2026 paper studied directional ablation against a frontier mixture-of-experts model — GLM-5.3-Flash, at "320B parameters, 288 routed experts, a four-wide hyper-connection residual, block-FP8". The technique, which the authors describe as "the canonical white-box attack on open-weight alignment", needs "no gradient-based training and no optimization, only a few hundred contrastive prompts". It had previously been demonstrated only on dense models up to roughly 70B.

It survived the shift to frontier MoE scale. The authors report reducing refusal behaviour by 77.6% through combined interventions across multiple weight modules — while noting that the standard recipe, applied naively, failed silently. "What it reaches is no longer where a reader of the original recipe would look for it."

Two things follow, and they point in different directions. If you self-host, the safety behaviour of the checkpoint is your responsibility, and you cannot assume the alignment is a durable property of the artefact. And if you are assessing risk from open-weight models generally, note that the attack is cheap, needs no training infrastructure, and now works at frontier scale.

If the reason you are self-hosting is to keep weights away from an infrastructure provider, the complementary control is hardware-level: what attestation does and does not prove is worth understanding before it appears in a design document as a solved problem.

When self-hosting actually makes sense

Four cases where the arithmetic tends to close.

Sustained high volume. API pricing is per token; self-hosting is per hour of reserved capacity. Above a utilisation threshold the fixed cost wins, and below it the fixed cost is mostly idle GPU. The threshold is higher than enthusiasm suggests, which is the recurring finding in UniverseBlend's breakdown of the hidden fees in a self-hosted bill.

Data that cannot leave. A genuine legal or contractual constraint, not a preference. This is the case where self-hosting is not competing on cost at all.

Latency floors. Where network round-trip time to a provider is a material fraction of your budget, and co-location changes the product.

Modification. Fine-tuning, distillation or architectural surgery you cannot perform through an API. Note that this case usually needs a smaller model than the flagship, which changes the memory arithmetic entirely and often makes the whole project viable.

What does not belong on the list is cost savings at low volume, control as a general sentiment, or the assumption that a 750B open-weight model is a drop-in substitute for a hosted frontier model. The first is usually false, the second is not a requirement, and the third is a hardware procurement exercise before it is a software one.

What this means going forward

The trend worth tracking is the widening gap between active and total parameters. When a 552B model runs 8B per token, the compute cost of serving frontier-quality output is falling fast while the memory cost is not. That pushes the binding constraint further toward HBM capacity and away from FLOPS, and it makes memory-per-dollar the number to watch in hardware announcements.

The second trend is quantised-native releases. Models shipping with FP8 or FP4 artefacts as first-class outputs rather than community conversions remove most of the quality uncertainty from quantisation, and effectively halve the hardware requirement for anyone willing to use them. That is the single change most likely to bring large open-weight models within reach of ordinary infrastructure.

Frequently asked questions

How much GPU memory do I need to run an open-weight model?

Start with parameters multiplied by bytes per parameter: two bytes at BF16, one at FP8, half at FP4. A 375B model needs about 750 GB at BF16 or 375 GB at FP8 for weights alone. Then add headroom for the KV cache and activations, sized against your realistic concurrent context length rather than the model's advertised maximum.

Does a mixture-of-experts model need less memory?

No — it needs less compute. Expert routing is decided per token at inference time, so every expert must stay resident in GPU memory. A model described as 375B-A23B stores 375B parameters and computes with 23B of them per token. Your hardware requirement follows the first number.

Is quantisation safe to use in production?

Increasingly yes, particularly where the provider ships a quantised artefact themselves rather than leaving it to the community. The caveat is that quality loss is uneven: aggregate scores move little while long-context recall and structured output adherence can move more. Evaluate on your own workload before committing.

Are open-weight licences all equivalent?

No. Current releases span MIT and Apache 2.0 — genuinely permissive — alongside bespoke model licences with their own conditions on commercial use, downstream training and use policies. "Open weights" describes availability, not licence terms, and the two are frequently confused.

Do open weights mean the model's safety training can be removed?

To a significant extent, yes, and cheaply. Published 2026 research demonstrates a training-free technique reducing refusal behaviour by 77.6% on a 320B parameter model using a few hundred contrastive prompts. If you deploy open weights, treat safety behaviour as something you verify in your own deployment rather than something inherited from the release.

Final takeaway

The licence tells you what you are permitted to do. The parameter count and the numeric format tell you what you can afford to do, and for most organisations the second constraint binds first and binds harder.

Do that arithmetic before the licence review, not after. Parameters times bytes per parameter, plus honest headroom for concurrent context, measured against the memory actually available in the hardware you can obtain. If that number does not close, nothing downstream of it matters — and if it does, the licence question becomes the straightforward one it was always assumed to be.

Sources and further reading

• GLM-5.3 model card — 753B parameters, MoE with dynamic sparse attention, 1M context
• DeepSeek-V4.1-Flash model card — 552B backbone, 8B/16B active, MIT licence
• K2-Horizon-375B-A23B model card — 375B stored, 23B per token, Apache 2.0
• Shi, Chen and Shen, How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE, arXiv 2609.09793
• NVIDIA GB200 NVL72 specifications — 72 Blackwell GPUs, 13.4 TB HBM3E