jietang (@jietang)

xcancel.com · 2026-08-19 · by Long Nguyen ·generic
Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

jietang (@jietang)

Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions. The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed. Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter. Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it. This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count. Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

Aug 19, 2026 · 5:04 AM UTC

75

202

1,523

100,568

Sort replies: Relevant Recent Liked

[](https://xcancel.com/FanShifu/status/2089953730657714233#m)

Replying to @jietang

唐老师，post-training 什么时候会有自己的 Chinchilla？预训练当年有 Kaplan 走偏后被纠正。也可能这颗旋钮的余量永远用不完，毕竟环境可以一直生成。很想知道 GLM 内部看到的 RL 曲线长什么样，是已经平了，还是仍有余力

2

1,075

[](https://xcancel.com/luyun0120/status/2089967148290707918#m)

Replying to @jietang

值得深思。这个问题背后有更深的结构，表面的答案往往只是冰山一角。能问出好问题的人，比能给出答案的人更稀缺。

3

1,217

[](https://xcancel.com/icehubin/status/2089978704692961590#m)

Replying to @jietang

Next time, maybe mid-training, pre-training, and even more. 唐老师说了，控制变量实验做完了，下次，都有。 方法论无敌！

573

[](https://xcancel.com/1501%5FSTUDIO/status/2089943079835062745#m)

Replying to @jietang

👏

1

1,243

[](https://xcancel.com/AllTheTokens/status/2089985650993115640#m)

Replying to @jietang

Solving complex problems is about about reducing and finding simplified state machines that can represent the complex problem in an easier to manipulate and search space. It's compression + search + manipulation + experimentation on the scale-invariant reasoning space.

2

319

[](https://xcancel.com/buswe%5Fcom/status/2089945156740841871#m)

Replying to @jietang

Justo, la arquitectura y la calidad de los datos mueven la aguja tanto como el recuento de parámetros, y salen mucho más baratas de escalar.

1

1,839

[](https://xcancel.com/kailunchen5/status/2089943191659467044#m)

Replying to @jietang

非常好的补充角度

1

1,206

[](https://xcancel.com/LanhuaX68906/status/2089957418054848598#m)

Replying to @jietang

楼下问 post-training 什么时候会有自己的 Chinchilla，GLM 的强化学习曲线是平了还是还有余量。唐杰说缩放不只是堆参数，GLM-5.3 和 5.2 参数一样，只多训了一个月长程环境和强化学习。

2

2,130

[](https://xcancel.com/hanqing1120/status/2089985599126348190#m)

Replying to @jietang

参数竞赛让位给旋钮选择。同底座只转后训练就出收益，这个控制实验比堆万亿参数更有说服力。

35

[](https://xcancel.com/nikhild791/status/2089970459882459555#m)

Replying to @jietang

Amazing insight 👏

703

[](https://xcancel.com/gokoururi628/status/2089942472952811549#m)

Replying to @jietang

深入浅出，对我来说很有启发，谢谢

2

991

[](https://xcancel.com/byteprobe/status/2089978137396564425#m)

Replying to @jietang

multi-axis compute scaling ftw! 🙌

508

[](https://xcancel.com/forlaisgroup/status/2089980529345179890#m)

Replying to @jietang

That's interesting, the parameter count says far less than what is actually packed into it, a thing we keep running into while getting 9x lossless compression with the contents still exact.

684

[](https://xcancel.com/iAmHenryMascot/status/2089957718966559200#m)

Replying to @jietang @realsigridjin

Great read! 5.3 is an awesome model It’s been pleasant to use so far! Thank you 🙏🏽

4

867

[](https://xcancel.com/MPxTrading/status/2089946366315561422#m)

Replying to @jietang

Cant wait for what you release next, thank you

1

1,838

[](https://xcancel.com/artists%5Fvoyage/status/2089959113916879179#m)

Replying to @jietang

GLM 5.3 shows that better post training can improve capability without simply making the model bigger

1

309

[](https://xcancel.com/malikwas1f/status/2089945321451450429#m)

Replying to @jietang

Thank you for this great insight

4

1,191

[](https://xcancel.com/wu89%5Fj/status/2089952104026640528#m)

Replying to @jietang

I am thinking an elephant has a bigger brain than a human. That doesn't mean an elephant is smarter than a human. Keep doing what you are doing. I will wait for good surprises from your group.

1

622

[](https://xcancel.com/ClaudeBadgez/status/2089948790040281319#m)

Replying to @jietang

Congratulations on your achievement, hoping you release smaller model with MOE. I feel smaller model from GLM 5.3 will be serious frontier model.

1

1,032

[](https://xcancel.com/MoonlitMonkey69/status/2089973173110661208#m)

Replying to @jietang

Benchmarks seem to suggest you are right. Hell, you are probably more right than you know, and even more can be packed inside of less parameters.

112

[](https://xcancel.com/NyanpasuKA/status/2089991462536520075#m)

Replying to @jietang

very clear and insightful thanks for sharing

[](https://xcancel.com/blueiswise/status/2089956188461170780#m)

Replying to @jietang

Enough talk! Just release GLM-5.5 😂. What we need now is vision capabilities, so we can close the loop on frontend development and testing. GLM is already doing a great job on the backend.😀

6

1,149

[](https://xcancel.com/limuvibecoding/status/2089948884089090418#m)

Replying to @jietang

Will glm5.5 be better than Fable5? I don't want to get completely crushed by Anthropic. Hope there's faster progress. (っ•̀ω•́)っ✎⁾⁾

9

1,935

[](https://xcancel.com/mrlag1114239/status/2089948293354700851#m)

Replying to @jietang

do humans use m-dashes?

1

858

[](https://xcancel.com/varun%5Fentropy/status/2089943154455953838#m)

Replying to @jietang

Congrats on GLM 5.3! If you'd kept scaling RL, would it still be getting better or were you hitting diminishing returns?

1

6

1,868

[](https://xcancel.com/nicolasembleton/status/2089948619004866638#m)

Replying to @jietang

Excellent read! Thanks for the insights. It seems Qwen 3.8 27B (and of course the excellent GLM 5.3) are a direct proof of your point. Density still has (had?) room for growth. Vision and speed are likely the next frontier for the GLM family. One question I've had on my mind for a while, do you think that not having vision enabled the model to perceive designs differently in its latent space? GLM 5.2 and 5.3 are \astonishing\ at design or even spatial senses. Since it doesn't have vision, did being born blind sharpen its other "senses" if you will?

1

13

2,432

[](https://xcancel.com/epsilver%5F/status/2089943082124886053#m)

Replying to @jietang

GLM 5.3’s performance is definitely impressive enough to show you’re right!!

8

1,070

[](https://xcancel.com/latentumx/status/2089971396697018703#m)

Replying to @jietang

We NEED data, LET ME scale

685

[](https://xcancel.com/MNX%5Ffi/status/2089987418556624944#m)

The AI race is no longer just a race to build the biggest model: @jietang argues that scaling is becoming multidimensional. The industry first scaled parameters aggressively, then Chinchilla showed that many models were under-trained relative to their size. As inference became a larger share of lifetime compute, the optimum shifted toward smaller models trained on far more data. MoE architectures added another dimension: total parameters can expand what a model knows, while activated parameters and effective depth influence how far it can reason. Now post-training is the new frontier. ZAI says GLM-5.3 uses the same base model, architecture, and parameter count as GLM-5.2. Spending a month scaling long-horizon environments and reinforcement learning produced substantial gains. The largest pre-training runs may be approaching diminishing returns, increasing the value of finding the next under-scaled axis: data, inference compute, effective depth, RL, environments, mid-training, or perhaps continual learning, if @SSI’s rumored bet proves right. The AI race is shifting from brutalist scaling toward multiple scaling frontiers, making the winner much harder to predict. ZAI / Zhipu is trading on MNX.fi Mainnet Beta.

MNX - The AI Exchange Trade Anthropic, memory stocks, and H100 prices with leverage on the first AI-focused decentralized futures exchange. mnx.fi

1

20

[](https://xcancel.com/vibecodeiro/status/2089951000186564614#m)

Replying to @jietang

Ok Jietang, agora barateie o plano do GLM

1

246

[](https://xcancel.com/AnasBf1/status/2089976577257791782#m)

Replying to @jietang

Amazing work and insights, thanks for open sourcing and for sharing your wisdom.

91

[](https://xcancel.com/dikakali/status/2089978782002417682#m)

Replying to @jietang

😀

35

[](https://xcancel.com/saurabh%5Fhere1/status/2089973604851286441#m)

Replying to @jietang

Where do algorithmic advancements fit into this? I have seen models converge much faster/with less data, using better loss formulations, RL, etc. Can we define a Pareto frontier without considering the algorithmic axis?

225

[](https://xcancel.com/thedeepflux/status/2089958696298357229#m)

Replying to @jietang

hmm, the emphasis on three is important. Parameter count became a lazy proxy because it is easy to quote, while tokens per parameter, training FLOPs, and evaluation loss explain whether the scale actually bought capability.

344

[](https://xcancel.com/zhou1234/status/2089963274809004215#m)

Replying to @jietang

多招人呀

62

[](https://xcancel.com/haibinLEE1/status/2089952867952255192#m)

Replying to @jietang

唐总对于RSI怎么看？LLM真的可以提出超出他的知识库的，前所未有的、范式级的创新吗？

2

181

Load more

[](https://xcancel.com/jietang/status/2089941544581403107#m)

xcancel.com · 36 images · 87 links

Comments (0)

No comments yet