Lossless AI compression: 33% less GPU memory, bit for bit.

getglyd.com · 2026-09-27 · by Long Nguyen ·generic
Glyd stores open models' bf16 weights in about 11 bits instead of 16 and decodes them inside the GPU's matrix multiply: bit for bit, often faster.

Lossless AI compression: 33% less GPU memory, bit for bit.

NewQwen3.8 27B, the top open model for one GPU, in 41 GB, bit for bit

Lossless AI compression: 33% less GPU memory, bit for bit.

Glyd stores the weights of open models like Llama, Qwen, Gemma and Mistral in about 11 bits instead of 16, and decodes them inside the GPU's matrix multiply. Every weight comes back exactly, so you run the same model on less hardware, and often faster.

Glyd stores the weights of open models in about 11 bits instead of 16 and decodes them inside the GPU's matrix multiply. The same model on less hardware, often faster.

Check your GPUHow it works

Qwen3-32B on an RTX A6000 (48 GB)Weights as Hugging Face lists them; a 48 GB GPU holds 51.5 GB. Run of Sep 26, 2026.

bf16

bf16, 65.5 GB: needs a second GPU

65.5 GBNeeds a second GPU

Glyd

Glyd, 44.5 GB: fits on one

44.5 GBFits: 43,338 of 49,140 MiB in use

48 GB GPU080 GB

The black line is the GPU's 48 GB. nvidia-smi, Sep 26, 2026.

32 to 33%smaller on every model measured, from SmolLM3 3B to Llama 4 Scout 109B.smaller on every model measured

Bit for bitEvery weight decodes to its exact bf16 value. Perplexity within 0.06% of bf16.every weight decodes to its exact value

3.4×the speed of DFloat11 at the same size, one sequence, Qwen3-8B on an RTX 4080 SUPER.DFloat11’s speed at one sequence

Up to 28%less GPU time a token than bf16: Qwen2.5-7B, 32 sequences, RTX 4080 SUPER.less GPU time a token than bf16

Measured on open models from

  • Qwen
  • Meta
  • Google
  • Mistral AI
  • DeepSeek
  • Hugging Face
  • Microsoft
  • Z.ai

    • Will it fit on your GPU?

    Pick your GPU's memory. Weights plus an 8K-token KV cache and 1.5 GB for the runtime, bf16 against Glyd.

    Weights, an 8K-token KV cache and the runtime, bf16 against Glyd.

    Updated Sep 27, 2026

    16 GBRTX 4080, RTX 508024 GBRTX 4090, RTX 309032 GBRTX 509048 GBRTX A6000, 6000 Ada80 GBH100, A10096 GBRTX PRO 6000, GH200

    | Model | bf16 | Glyd | Memory it needs, against this GPU | On one 24 GB GPU | | ------------------------------------------------------------------------------------------------------------------------ | ------- | ------- | --------------------------------- | ----------------- | | Gemma 3 12Best. | 24.4 GB | 16.4 GB | Fits only with Glyd | 24.4 GB → 16.4 GB | | Phi-4 14Best. | 29.3 GB | 19.7 GB | Fits only with Glyd | 29.3 GB → 19.7 GB | | R1 Distill Qwen 14Best. | 29.5 GB | 20.1 GB | Fits only with Glyd | 29.5 GB → 20.1 GB | | Qwen3 8Best. | 16.4 GB | 11.2 GB | Fits either way | 16.4 GB → 11.2 GB | | Llama 3.1 8Best. | 16.1 GB | 10.8 GB | Fits either way | 16.1 GB → 10.8 GB | | Mistral 7B v0.3est. | 14.5 GB | 9.8 GB | Fits either way | 14.5 GB → 9.8 GB | | Mistral Small 3.2 24Best. | 48.0 GB | 32.2 GB | Too big for one | 48.0 GB → 32.2 GB | | Gemma 4 26B-A4Best. | 51.6 GB | 34.7 GB | Too big for one | 51.6 GB → 34.7 GB |

    bf16Glydthis GPUSizes measured; est. means worked out from the parameter count, run queued.All models

    Smaller, and usually faster

    Generating a token reads every weight once, so reading a third fewer bytes saves time. At many sequences a step on A100 and H100 the decode work shows, and Glyd is slower there today.

    All benchmarks

    | GPU time a token against bf16, Qwen2.5-7B | 1 sequence | 32 sequences | | ----------------------------------------- | ------------- | ------------- | | RTX 4080 SUPER | 25% less time | 28% less time | | A10 | 28% less time | 18% less time | | A100 | 17% less time | 6% more time | | H100 SXM | 5% less time | 8% more time |

    Decode steps only, q, k, v and gate, up merged as vLLM runs them. Sep 26, 2026.

    How it works, in three stepsHow it works

    The full explanation

    1. 01

    Find the waste

    A bf16 weight spends 8 bits on its exponent, which carries about 2.6 bits of information in a trained model.A bf16 weight spends 8 bits on its exponent, which carries about 2.6 bits of information. 2. 02

    Code it exactly

    Common exponents get short codes and rare ones a side list. Sign and mantissa stay as they are: about 11 bits a weight, nothing rounded.Short codes for common exponents, a side list for rare ones. Nothing rounded. 3. 03

    Decode in the multiply

    The GPU reads the compressed bytes and decodes them in registers, straight into the tensor cores. No bf16 copy is ever made.Decoded in registers, straight into the tensor cores. No bf16 copy in memory. The full explanation

    How it compares

    Lossless weight compression is an active research area. Here is where Glyd sits, on numbers we measured ourselves where we could.

    | Method | Bits a weight | Same model? | Where the weights are decoded | | ------------------------------------ | ------------- | ---------------- | ---------------------------------------------------------------------------- | | GlydOpen source, 2026 | 10.8 or 12.0 | Yes, bit for bit | Inside the matrix multiply, in registers. 3.4× DFloat11 at one sequence. | | DFloat11NeurIPS 2025 | about 11 | Yes, bit for bit | Each block decompressed to bf16 in memory before it runs. | | ZipServASPLOS 2026 | 11.35 | Yes, bit for bit | Inside the matrix multiply. Its kernels and Glyd’s are within a few percent. | | 4-bit quantizationGGUF Q4, AWQ, GPTQ | about 4.5 | No, rounded | Smallest by far, and a different model: its answers change. |

    Latest

    All reports

  • Report · Sep 27, 2026Nineteen open models: every matrix 32 to 33% smaller, bit for bitEvery Linear layer's matrix of nineteen open models, SmolLM3 3B to GLM-4.5-Air, packed and unpacked bit for bit: 31.9 to 33.0% smaller, quality on ten.
  • Report · Sep 27, 2026Qwen3.8 27B, the top open model for one GPU, in 41 GBQwen3.8 27B, the top-scoring open model that fits one GPU, in 41,071 MiB of GPU memory with Glyd against bf16's 51,771: under a 48 GB card, bit for bit.
  • [Plans · Sep 27, 2026What's next for Glyd: pip install, vLLM, one commandFrom a PyTorch harness to pip install "glyd\[gpu\]" with models ready on Hugging Face, then serving through vLLM, then glyd fit, pack and serve. No dates.](https://getglyd.com/blog/whats-next-pip-install/)

Questions

All questions

Is this quantization?

No. Quantization rounds weights to fewer levels, which changes the model. Glyd stores the same values in fewer bits and gives every one of them back. It is closer to a zip file than to 4-bit.

Does it change the model's answers?

The weights are the model's to the bit. The matrix products add their terms in another order than cuBLAS does, as any two GPU kernels do, so a close call can go either way: over the models measured, Glyd's MMLU answers are bf16's on 98.0 to 100% of the questions and perplexity stays within 0.06% of bf16's.

Which GPUs?

Measured on an RTX 4080 SUPER, RTX A6000s, an A10, an A100 and H100s. Glyd picks the layout for the GPU: the tiered one on RTX 40-series (Ada) and wherever only it fits, the 12-bit one on the others. Blackwell GPUs are not measured yet.

How do I install it?

Today Glyd on the GPU runs from its PyTorch harness: clone the repository, run gpu/setup\_env.sh, then e2e.py on a model. A Python package, pip install "glyd\[gpu\]" with glyd.from\_pretrained(), is in progress. The codec underneath installs with Homebrew, cargo or a wheel from the releases.

Try it on your model

Point the harness at any model released in bf16. It packs the weights, checks every one bit for bit, and runs bf16 and Glyd side by side on your GPU.

Check your GPUView on GitHub

git clone https://github.com/surya-koritala/Glyd
cd Glyd/gpu
python e2e.py /models/Llama-3.1-8B-Instruct \
  --format auto --fused --baseline --batch 1,8,32
getglyd.com · 9 images · 32 links

Comments (0)

No comments yet