@rileybrown: If I had a budget of $10,000 to build a local setup to run GLM 5.2 locally AND any future local model that comes out soo…

Replies (11)

Ludo @lludol

I don't believe it's enough

Riley Brown @rileybrown

How much?

Austin S. Lin @siraustin

2 x 128GB GB10

Ludo @lludol

How much this cost?

Austin S. Lin @siraustin

$8-10k in U.S.

Ludo @lludol

But at what the models run? I heard it's pretty slow (compare to frontier speed or just same model via API) with this budget

Austin S. Lin @siraustin

ds4 flash — I was getting 20-25 t/s but apparently latest vLLM recipes get 45 t/s up to 1m context

Gumbii.Digital @GumbiiDigital

That’s 2bit and it runs like absolute shit. You need 8x 128GB to make it run comfortably and be useful.

Austin S. Lin @siraustin

I think it will be done in 4x

Gumbii.Digital @GumbiiDigital

I think so too

Austin S. Lin @siraustin

if you had a time machine, 512GB M3 Ultra

Comments (0)