Gemini 3.7 Flash, Grok 4.6, GLM-5.3 and DeepSeek V4 Pro joined the frontier

This mid August is as hot as July when it comes to LLM releases: Grok 4.6, Gemini 3.7 Flash, DeepSeek V4 Pro 0813 as well as open-weight models: Qwen3.8 Max, Qwen3.8 27B, and GLM-5.3.

Intelligence Index vs. Cost per Intelligence Index Taskfrom Artificial Analysis. Note that it is based on score of benchmarks like Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond - not necessarily fluid intelligence like in abstract puzzle games of ARC-AGI-3 or Baba is You.

Since these models are smart, I decided to rerun the Baba Is Bench, to see how the models fare on a puzzle game. Even though the game is popular, we check for spoilers - and to our surprise, there are no signs of models knowing solutions ahead - unlike in SWE-bench Verified.

Results are stunning for Grok, Gemini Flash or DeepSeek. Don’t get deceived by minor version changes - they all give qualitatively different results! For both GLM and Qwen the progress was more gradual.

As previously, we do it in two rounds. First, asking to play the initial stage the Intro, then for those who pass at least 7/8 levels, also the next one the Lake.

Stage 0: The Intro[](https://quesma.com/blog/baba-is-aug-2026/#stage-0-the-intro)

We show the models we analyze, against dimmed previous ones.

Download as PNG

| Model | 00 baba is you | 01 where do i go? | 02 now what is this? | 03 out of reach | 04 still out of reach | 05 volcano | 06 off limits | 07 grass yard | pass@1 ↓ | pass@3 ↓ | Turns ↓ | Output tokens ↓ | Total cost ↑ | | ------------------------------ | -------------- | ----------------- | -------------------- | --------------- | --------------------- | ---------- | ------------- | ------------- | -------- | -------- | ------- | --------------- | ------------ | | GoogleGemini 3.7 Flash | | | | | | | | | 100% | 100% | 53 | 22k | $5.38 | | AnthropicClaude Opus 5 | | | | | | | | | 100% | 100% | 7 | 23k | $17.15 | | OpenAIGPT-5.5 | | | | | | | | | 100% | 100% | 8 | 15k | $13.62 | | AnthropicClaude Fable 5 | | | | | | | | | 100% | 100% | 6 | 29k | $40.98 | | OpenAIGPT-5.6 Sol | | | | | | | | | 100% | 100% | 11 | 10k | $11.91 | | GrokGrok 4.6 | | | | | | | | | 96% | 100% | 25 | 52k | $14.01 | | Z.aiGLM-5.3 | | | | | | | | | 96% | 100% | 13 | 47k | $6.20 | | KimiKimi K3 | | | | | | | | | 96% | 100% | 9 | 29k | $12.56 | | AnthropicClaude Opus 4.8 | | | | | | | | | 96% | 100% | 35 | 46k | $63.14 | | Z.aiGLM-5.2 | | | | | | | | | 96% | 100% | 12 | 75k | $6.20 | | GoogleGemini 3.1 Pro | | | | | | | | | 92% | 100% | 8 | 38k | $12.41 | | GoogleGemini 3.6 Flash | | | | | | | | | 88% | 100% | 92 | 82k | $124.31 | | OpenAIGPT-5.6 Terra | | | | | | | | | 88% | 100% | 41 | 55k | $34.28 | | DeepSeekDeepSeek V4 Pro 0813 | | | | | | | | | 75% | 88% | 18 | 130k | $2.94 | | DeepSeekDeepSeek V4 Flash | | | | | | | | | 75% | 100% | 55 | 190k | $2.82 | | GrokGrok 4.5 | | | | | | | | | 75% | 88% | 86 | 68k | $50.66 | | DeepSeekDeepSeek V4 Flash 0731 | | | | | | | | | 75% | 88% | 22 | 156k | $1.16 | | AnthropicClaude Sonnet 5 | | | | | | | | | 67% | 88% | 22 | 151k | $42.21 | | OpenAIGPT-5.6 Luna | | | | | | | | | 63% | 88% | 118 | 129k | $59.32 | | QwenQwen3.8 Max | | | | | | | | | 58% | 75% | 9 | 66k | $10.75 | | QwenQwen3.6 27B | | | | | | | | | 29% | 38% | 24 | 82k | $5.59 | | DeepSeekDeepSeek V4 Pro | | | | | | | | | 29% | 38% | 37 | 183k | $11.49 | | QwenQwen3.7 Max | | | | | | | | | 25% | 38% | 21 | 132k | $16.98 | | QwenQwen3.8 27B | | | | | | | | | 21% | 38% | 9 | 134k | $9.07 |

passtimeagent stepsmovessolvedtimeoutwrong answernot solved

The results are nothing short of stunning, as both results went up and prices went down significantly.

• Gemini 3.7 Flash now solves every trial, and is over 20x cheaper than its predecessor, Gemini 3.6 Flash.
• Grok 4.6 solved all but one attempt (vs Grok 4.5 that missed 6 attempts and one level), while being over 3x cheaper.
• DeepSeek V4 Pro 0813 solved 7/8 levels vs its previous version that solved only 3/8 - and while being 4x cheaper.

Download as PNG

0%25%50%75%100%$1$2$5$10$20$50$100cost of the full intro run, log scalepass@1AnthropicOpenAIAnthropicOpenAIKimiAnthropicZ.aiGoogleOpenAIDeepSeekAnthropicOpenAIQwenQwenQwenQwenGoogleGemini 3.6 FlashGoogleGemini 3.7 FlashGrokGrok 4.5GrokGrok 4.6DeepSeekDeepSeek V4 ProDeepSeekDeepSeek V4 Pro 0813DeepSeekDeepSeek V4 Flash 0731Z.aiGLM-5.3

With the GLM and Qwen family there is less success:

• GLM-5.3 got exactly the same result as GLM-5.2.
• Qwen3.8 Max is better than Qwen3.7 Max (way more attempts solved), but still far from completing all levels.
• Qwen3.8 27B has worse results than Qwen3.6 27B - though, all within experimental noise.

I was surprised that it didn’t do better. Especially as Qwen3.8 27B scores 52 on Artificial Analysis. From what I’ve seen in the transcripts, it tried to solve in one go, without actually exploring the board. It matches Simon Willison’s experience that Qwen3.8 27B defaults to wildly overthinking things. While Baba Is You clearly needs thinking, it is a balance of thinking and testing. Compare and contrast with previous Grok and Gemini Flash models that were doing a lot and thinking too little.

Download as PNG

$1$2$5$10$20$50$1005102050100agent turns (mean per attempt)\ one level unsolvedtotal cost← more deliberatemore trigger-happy →← cheapermore expensive →GoogleGemini 3.7 FlashAnthropicClaude Opus 5OpenAIGPT-5.5AnthropicClaude Fable 5OpenAIGPT-5.6 SolGrokGrok 4.6Z.aiGLM-5.3KimiKimi K3AnthropicClaude Opus 4.8Z.aiGLM-5.2GoogleGemini 3.1 ProGoogleGemini 3.6 FlashOpenAIGPT-5.6 TerraDeepSeekDeepSeek V4 Pro 0813\*DeepSeekDeepSeek V4 FlashGrokGrok 4.5\*DeepSeekDeepSeek V4 Flash 0731\*AnthropicClaude Sonnet 5\*OpenAIGPT-5.6 Luna\

It is interesting to look at the chart of agent turns and price. Grok and Gemini Flash moved from trigger-happy models, to more moderate ones, resulting in a significantly reduced cost.

Stage 1: The Lake[](https://quesma.com/blog/baba-is-aug-2026/#stage-1-the-lake)

For models that passed at least 7 out of 8 intro levels, we run them on the next stage, the Lake. Only one attempt per level, as it is costly. Or at least - it used to be costly with the previous models, as now it is getting significantly better.

| Model | Harness | | | | | | | | | | | | | | | | Solved | Total time | Output tokens | Total cost | | ------------------------------ | ---------------------- | ---------------------------------------------------- | ----------------------------------------------- | ----------------------------------------------- | ---------------------------------------------------- | ------------------------------------------------ | ---------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------ | ----------------------------------------------------- | ------------------------------------------------------ | ------------------------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------- | ------ | ---------- | ------------- | ---------- | | AnthropicClaude Fable 5 | Claude CodeClaude Code | 01 Icy Waters solved · 4m · 277k tokens · $1.73 | 02 Turns solved · 11m · 1.1M tokens · $4.83 | 03 Affection solved · 3m · 229k tokens · $0.99 | 04 Pillar Yard solved · 8m · 777k tokens · $3.64 | 05 Brick Wall solved · 2m · 206k tokens · $0.67 | 06 Lock solved · 6m · 795k tokens · $2.76 | 07 Novice Locksmith solved · 6m · 695k tokens · $2.46 | 08 Locked In solved · 5m · 396k tokens · $1.70 | 09 Changeless solved · 3m · 213k tokens · $1.37 | 10 Two Doors solved · 16m · 1.8M tokens · $5.96 | 11 Jelly Throne solved · 11m · 570k tokens · $3.66 | 12 Crab Storage solved · 12m · 1.5M tokens · $5.00 | 13 Burglary solved · 10m · 1.1M tokens · $4.52 | b1 Submerged Ruins solved · 11m · 676k tokens · $3.54 | b2 Sunken Temple stopped · 49m · 337k tokens · $10.41 | 14 | 2h 36m | 10.7M | $53.25 | | AnthropicClaude Opus 5 | Claude CodeClaude Code | 01 Icy Waters solved · 3m · 365k tokens · $0.60 | 02 Turns solved · 16m · 940k tokens · $2.33 | 03 Affection solved · 7m · 596k tokens · $0.83 | 04 Pillar Yard solved · 13m · 948k tokens · $1.88 | 05 Brick Wall solved · 5m · 255k tokens · $0.38 | 06 Lock solved · 10m · 783k tokens · $1.46 | 07 Novice Locksmith solved · 8m · 845k tokens · $1.05 | 08 Locked In solved · 10m · 988k tokens · $1.43 | 09 Changeless solved · 6m · 345k tokens · $0.52 | 10 Two Doors solved · 45m · 2.8M tokens · $7.26 | 11 Jelly Throne solved · 8m · 725k tokens · $1.49 | 12 Crab Storage solved · 1h 33m · 3.2M tokens · $16.45 | 13 Burglary solved · 12m · 862k tokens · $1.81 | b1 Submerged Ruins solved · 15m · 999k tokens · $2.34 | b2 Sunken Temple stopped · 2h 35m · 1.7M tokens · $22.12 | 14 | 6h 46m | 16.3M | $61.96 | | GoogleGemini 3.7 Flash | Terminus-2 | 01 Icy Waters solved · 7m · 3.4M tokens · $0.34 | 02 Turns solved · 12m · 9.9M tokens · $0.75 | 03 Affection solved · 3m · 520k tokens · $0.09 | 04 Pillar Yard solved · 4m · 428k tokens · $0.08 | 05 Brick Wall solved · 14m · 9.5M tokens · $0.74 | 06 Lock solved · 20m · 23.7M tokens · $1.48 | 07 Novice Locksmith solved · 4m · 627k tokens · $0.12 | 08 Locked In solved · 5m · 1.4M tokens · $0.19 | 09 Changeless solved · 6m · 1.4M tokens · $0.20 | 10 Two Doors solved · 2h 47m · 385.5M tokens · $17.98 | 11 Jelly Throne solved · 9m · 2.9M tokens · $0.34 | 12 Crab Storage solved · 34m · 37.0M tokens · $2.10 | 13 Burglary solved · 11m · 4.6M tokens · $0.45 | b1 Submerged Ruins solved · 15m · 6.1M tokens · $0.53 | b2 Sunken Temple failed · 4h 0m · 536.0M tokens · $30.36 | 14 | 9h 11m | 1023.1M | $55.76 | | OpenAIGPT-5.6 Sol | CodexCodex | 01 Icy Waters solved · 2m · 100k tokens · $0.29 | 02 Turns solved · 16m · 332k tokens · $0.72 | 03 Affection solved · 4m · 87k tokens · $0.28 | 04 Pillar Yard solved · 1h 18m · 5.4M tokens · $6.24 | 05 Brick Wall solved · 6m · 99k tokens · $0.31 | 06 Lock solved · 24m · 358k tokens · $0.97 | 07 Novice Locksmith solved · 11m · 222k tokens · $0.58 | 08 Locked In solved · 7m · 135k tokens · $0.33 | 09 Changeless solved · 2m · 196k tokens · $0.15 | 10 Two Doors solved · 36m · 1.3M tokens · $2.72 | 11 Jelly Throne solved · 27m · 712k tokens · $1.51 | 12 Crab Storage solved · 49m · 1.9M tokens · $4.06 | 13 Burglary solved · 24m · 935k tokens · $2.33 | b1 Submerged Ruins solved · 37m · 432k tokens · $1.73 | b2 Sunken Temple failed · 3h 53m · 4.2M tokens · $9.19 | 14 | 9h 16m | 16.4M | $31.42 | | OpenAIGPT-5.5 | CodexCodex | 01 Icy Waters solved · 16m · 2.3M tokens · $3.34 | 02 Turns solved · 26m · 2.1M tokens · $5.63 | 03 Affection solved · 6m · 246k tokens · $0.86 | 04 Pillar Yard solved · 33m · 550k tokens · $4.48 | 05 Brick Wall solved · 8m · 326k tokens · $1.14 | 06 Lock solved · 30m · 1.3M tokens · $5.35 | 07 Novice Locksmith solved · 5m · 295k tokens · $0.95 | 08 Locked In solved · 5m · 375k tokens · $0.90 | 09 Changeless solved · 3m · 192k tokens · $0.65 | 10 Two Doors solved · 1h 19m · 5.3M tokens · $13.58 | 11 Jelly Throne solved · 49m · 1.4M tokens · $8.16 | 12 Crab Storage failed · 1h 11m · 16.5M tokens · $17.66 | 13 Burglary solved · 44m · 6.3M tokens · $10.33 | b1 Submerged Ruins solved · 1h 38m · 9.2M tokens · $20.13 | b2 Sunken Temple failed · 1h 4m · 3.8M tokens · $8.92 | 13 | 8h 57m | 50.1M | $102.08 | | GrokGrok 4.6 | Terminus-2 | 01 Icy Waters solved · 21m · 340k tokens · $0.66 | 02 Turns solved · 45m · 1.7M tokens · $2.15 | 03 Affection solved · 4m · 85k tokens · $0.15 | 04 Pillar Yard solved · 54m · 1.3M tokens · $2.01 | 05 Brick Wall solved · 6m · 113k tokens · $0.16 | 06 Lock solved · 9m · 532k tokens · $0.49 | 07 Novice Locksmith solved · 39m · 2.5M tokens · $2.17 | 08 Locked In solved · 8m · 523k tokens · $0.64 | 09 Changeless solved · 14m · 489k tokens · $0.53 | 10 Two Doors solved · 2h 52m · 7.7M tokens · $7.60 | 11 Jelly Throne solved · 1h 18m · 1.4M tokens · $2.38 | 12 Crab Storage solved · 3h 41m · 9.4M tokens · $9.45 | 13 Burglary solved · 1h 4m · 3.8M tokens · $3.61 | b1 Submerged Ruins failed · 4h 0m · 5.2M tokens · $7.50 | b2 Sunken Temple failed · 4h 0m · 3.6M tokens · $6.71 | 13 | 20h 16m | 38.6M | $46.20 | | DeepSeekDeepSeek V4 Pro 0813 | Claude CodeClaude Code | 01 Icy Waters solved · 2h 18m · 2.8M tokens · $0.39 | 02 Turns solved · 46m · 4.3M tokens · $0.38 | 03 Affection solved · 10m · 488k tokens · $0.06 | 04 Pillar Yard solved · 1h 26m · 3.2M tokens · $0.33 | 05 Brick Wall solved · 11m · 820k tokens · $0.09 | 06 Lock solved · 25m · 3.7M tokens · $0.32 | 07 Novice Locksmith solved · 27m · 2.9M tokens · $0.22 | 08 Locked In solved · 18m · 1.4M tokens · $0.10 | 09 Changeless solved · 15m · 822k tokens · $0.08 | 10 Two Doors solved · 3h 35m · 16.9M tokens · $2.25 | 11 Jelly Throne solved · 55m · 6.6M tokens · $0.56 | 12 Crab Storage failed · 4h 1m · 15.0M tokens · $1.95 | 13 Burglary solved · 2h 42m · 8.8M tokens · $0.78 | b1 Submerged Ruins solved · 2h 4m · 2.6M tokens · $0.36 | b2 Sunken Temple failed · 2h 35m · 5.3M tokens · $0.62 | 13 | 22h 7m | 75.6M | $8.48 | | AnthropicClaude Opus 4.8 | Claude CodeClaude Code | 01 Icy Waters failed · 6m · 445k tokens · $1.15 | 02 Turns solved · 49m · 5.5M tokens · $9.57 | 03 Affection solved · 8m · 732k tokens · $1.41 | 04 Pillar Yard solved · 1h 2m · 4.6M tokens · $10.96 | 05 Brick Wall solved · 4m · 469k tokens · $0.70 | 06 Lock solved · 17m · 2.1M tokens · $3.81 | 07 Novice Locksmith solved · 9m · 1.2M tokens · $1.75 | 08 Locked In solved · 9m · 989k tokens · $1.80 | 09 Changeless solved · 10m · 1.0M tokens · $1.79 | 10 Two Doors solved · 28m · 3.3M tokens · $5.20 | 11 Jelly Throne solved · 44m · 5.2M tokens · $9.01 | 12 Crab Storage failed · 1h 46m · 7.1M tokens · $23.69 | 13 Burglary solved · 53m · 5.6M tokens · $9.53 | b1 Submerged Ruins solved · 1h 34m · 10.5M tokens · $20.25 | b2 Sunken Temple failed · 47m · 1.1M tokens · $7.15 | 12 | 9h 7m | 49.9M | $107.76 | | Z.aiGLM-5.3 | Terminus-2 | 01 Icy Waters solved · 50m · 207k tokens · $0.50 | 02 Turns solved · 1h 13m · 324k tokens · $0.81 | 03 Affection solved · 17m · 75k tokens · $0.16 | 04 Pillar Yard solved · 1h 26m · 396k tokens · $0.95 | 05 Brick Wall solved · 5m · 18k tokens · $0.04 | 06 Lock solved · 38m · 175k tokens · $0.39 | 07 Novice Locksmith solved · 1h 8m · 348k tokens · $0.77 | 08 Locked In solved · 16m · 79k tokens · $0.16 | 09 Changeless solved · 14m · 95k tokens · $0.14 | 10 Two Doors solved · 3h 26m · 2.1M tokens · $2.87 | 11 Jelly Throne solved · 1h 45m · 691k tokens · $1.24 | 12 Crab Storage failed · 4h 1m · 2.7M tokens · $3.01 | 13 Burglary solved · 1h 39m · 619k tokens · $1.17 | b1 Submerged Ruins failed · 4h 1m · 772k tokens · $2.19 | b2 Sunken Temple failed · 4h 1m · 954k tokens · $2.44 | 12 | 24h 57m | 9.6M | $16.85 | | Z.aiGLM-5.2 | Claude CodeClaude Code | 01 Icy Waters solved · 49m · 1.7M tokens · $1.35 | 02 Turns solved · 2h 15m · 8.3M tokens · $4.11 | 03 Affection solved · 14m · 395k tokens · $0.30 | 04 Pillar Yard solved · 1h 22m · 1.8M tokens · $1.61 | 05 Brick Wall solved · 9m · 249k tokens · $0.17 | 06 Lock solved · 35m · 1.5M tokens · $0.96 | 07 Novice Locksmith solved · 31m · 1.7M tokens · $1.27 | 08 Locked In solved · 42m · 1.2M tokens · $0.80 | 09 Changeless solved · 14m · 739k tokens · $0.89 | 10 Two Doors failed · 2h 46m · 6.2M tokens · $4.02 | 11 Jelly Throne failed · 4h 27m · 17.0M tokens · $7.89 | 12 Crab Storage solved · 1h 27m · 3.4M tokens · $2.39 | 13 Burglary solved · 1h 26m · 3.6M tokens · $2.19 | b1 Submerged Ruins failed · 4h 33m · 11.4M tokens · $7.29 | b2 Sunken Temple failed · 2h 11m · 3.8M tokens · $2.29 | 11 | 23h 41m | 62.9M | $37.51 | | KimiKimi K3 | kimi-cli | 01 Icy Waters solved · 25m · 222k tokens · $0.81 | 02 Turns failed · 4h 1m · 29.2M tokens · $14.27 | 03 Affection solved · 25m · 1.1M tokens · $1.07 | 04 Pillar Yard failed · 38m · 704k tokens · $1.41 | 05 Brick Wall solved · 5m · 76k tokens · $0.17 | 06 Lock solved · 2h 51m · 16.6M tokens · $8.45 | 07 Novice Locksmith solved · 45m · 2.6M tokens · $2.19 | 08 Locked In solved · 1h 37m · 10.2M tokens · $5.50 | 09 Changeless solved · 55m · 5.3M tokens · $2.98 | 10 Two Doors solved · 3h 36m · 13.8M tokens · $8.98 | 11 Jelly Throne solved · 1h 2m · 379k tokens · $1.83 | 12 Crab Storage solved · 3h 0m · 14.2M tokens · $8.58 | 13 Burglary solved · 2h 41m · 21.1M tokens · $10.65 | b1 Submerged Ruins failed · 4h 0m · 4.1M tokens · $4.80 | b2 Sunken Temple failed · 3h 33m · 10.0M tokens · $6.46 | 11 | 29h 34m | 129.5M | $78.15 | | GoogleGemini 3.1 Pro | Terminus-2 | 01 Icy Waters solved · 4m · 54k tokens · $0.39 | 02 Turns failed · 9m · 96k tokens · $1.04 | 03 Affection solved · 3m · 74k tokens · $0.34 | 04 Pillar Yard solved · 10m · 104k tokens · $1.06 | 05 Brick Wall solved · 2m · 22k tokens · $0.16 | 06 Lock failed · 53m · 597k tokens · $5.54 | 07 Novice Locksmith solved · 4m · 49k tokens · $0.41 | 08 Locked In solved · 7m · 79k tokens · $0.69 | 09 Changeless solved · 4m · 62k tokens · $0.44 | 10 Two Doors failed · 14m · 145k tokens · $1.54 | 11 Jelly Throne failed · 35m · 369k tokens · $3.61 | 12 Crab Storage failed · 8m · 76k tokens · $0.83 | 13 Burglary failed · 15m · 149k tokens · $1.57 | b1 Submerged Ruins solved · 49m · 533k tokens · $4.96 | b2 Sunken Temple failed · 2h 20m · 39.4M tokens · $35.59 | 8 | 5h 58m | 41.8M | $58.17 | | GoogleGemini 3.6 Flash | Terminus-2 | 01 Icy Waters stopped · 38m · 106.2M tokens · $25.94 | 02 Turns failed · 52m · 31.6M tokens · $10.16 | 03 Affection solved · 4m · 896k tokens · $0.86 | 04 Pillar Yard solved · 20m · 22.4M tokens · $7.70 | 05 Brick Wall solved · 3m · 398k tokens · $0.51 | 06 Lock stopped · 55m · 207.8M tokens · $44.35 | 07 Novice Locksmith stopped · 58m · 192.4M tokens · $43.04 | 08 Locked In solved · 4m · 508k tokens · $0.54 | 09 Changeless solved · 2m · 72k tokens · $0.10 | 10 Two Doors stopped · 54m · 141.2M tokens · $35.11 | 11 Jelly Throne stopped · 6m · 1.2M tokens · $1.13 | 12 Crab Storage solved · 7m · 1.2M tokens · $1.38 | 13 Burglary stopped · 58m · 151.3M tokens · $35.70 | b1 Submerged Ruins stopped · 58m · 150.1M tokens · $35.69 | b2 Sunken Temple stopped · 47m · 110.9M tokens · $27.29 | 6 | 7h 46m | 1118.1M | $269.48 | | DeepSeekDeepSeek V4 Flash 0731 | Claude CodeClaude Code | 01 Icy Waters solved · 30m · 2.4M tokens · $0.12 | 02 Turns solved · 3h 49m · 17.2M tokens · $1.28 | 03 Affection failed · 35m · 1.6M tokens · $0.04 | 04 Pillar Yard failed · 1h 40m · 4.3M tokens · $0.31 | 05 Brick Wall solved · 22m · 649k tokens · $0.03 | 06 Lock failed · 10m · 50k tokens · $0.00 | 07 Novice Locksmith solved · 4h 57m · 20.8M tokens · $0.64 | 08 Locked In solved · 36m · 4.8M tokens · $0.19 | 09 Changeless solved · 38m · 1.7M tokens · $0.07 | 10 Two Doors failed · 1h 28m · 9.2M tokens · $0.45 | 11 Jelly Throne failed · 1h 39m · 8.6M tokens · $0.32 | 12 Crab Storage failed · 1h 17m · 3.4M tokens · $0.15 | 13 Burglary failed · 1h 30m · 3.4M tokens · $0.10 | b1 Submerged Ruins failed · 23m · 1.2M tokens · $0.04 | b2 Sunken Temple failed · 4h 36m · 13.5M tokens · $0.55 | 6 | 24h 11m | 92.8M | $4.31 |

solvedstoppedfailed

Time[](https://quesma.com/blog/baba-is-aug-2026/#time)

First, let’s look at the wall time, for models run via APIs. While it might change with providers, or model bandwidth at a given time, it gives a ballpark estimate of effective model speed. In everyday software engineering it is what we care for, anyway.

Download as PNG

0m3m10m30m1h2h4h6h8h10h12h14h16h18h02468101214cumulative wall clocklinear·loglevels solvedGoogleGrokDeepSeekDeepSeekAnthropicKimiGoogleOpenAIOpenAIAnthropicAnthropicZ.aiZ.aiGoogleGemini 3.7 FlashGrok 4.6DeepSeek V4 Pro 0813DeepSeek V4 Flash 0731Claude Opus 5Kimi K3Gemini 3.6 FlashGPT-5.6 SolGPT-5.5Claude Fable 5Claude Opus 4.8GLM-5.2GLM-5.3Gemini 3.1 Pro

And wow, Gemini 3.7 Flash is fast - not only per token (which is widely cited but largely irrelevant), but per problem solved. Here, for most levels, only Claude Fable 5 is faster.

Grok 4.6, while 2x slower than GPT-5.6 Sol, still manages to solve 13 of 15 levels (GPT-5.6 Sol solves 14). It’s a promotion to the league of top models.

DeepSeek V4 Pro 0813 might be the slowest of the top models - but it is the first open-weight model that finally reached the same top league. It is way beyond GLM-5.2 or Kimi K3 results.

GLM-5.3 was almost the same as GLM-5.2, to the point that two lines overlap; though, it solved one more level.

Cost[](https://quesma.com/blog/baba-is-aug-2026/#cost)

Now let’s look at costs. It’s beautiful.

Download as PNG

$0$0.3$1$3$10$20$30$40$60$8002468101214cumulative costlinear·loglevels solvedGoogleGrokDeepSeekDeepSeekAnthropicKimiGoogleOpenAIOpenAIAnthropicAnthropicZ.aiZ.aiGoogleGemini 3.7 FlashGrok 4.6DeepSeek V4 Pro 0813DeepSeek V4 Flash 0731Claude Opus 5Kimi K3Gemini 3.6 FlashGPT-5.6 SolGPT-5.5Claude Fable 5Claude Opus 4.8GLM-5.2GLM-5.3Gemini 3.1 Pro

Grok 4.6 is cheaper than Fable 5, but more expensive than GPT-5.6 Sol or Opus 5.

While Gemini 3.6 Flash was ridiculously expensive, Gemini 3.7 Flash is one of the cheapest models. At 12 levels, it is twice as cheap as GPT-5.6 Sol, the previous cheapest model. At 13 tasks it is still twice as cheap. Only vs DeepSeek Pro is it slightly more expensive.

What happens with DeepSeek V4 Pro 0813 is a pure wonder. Not only is it open-weight, but also the cheapest of all.

GLM-5.3 was 2x cheaper than its predecessor.

Some lessons about harnesses[](https://quesma.com/blog/baba-is-aug-2026/#some-lessons-about-harnesses)

In previous experiments we used various harnesses. Here we used Harbor\-default Terminus-2, except DeepSeek, which we ran on Claude Code. While there are Grok Build and DeepSeek Harness, results were slightly worse (Grok) or vastly inferior (DeepSeek). That said, these are newer tools, and likely to get improved in the future. If there is interest, we’re happy to share more insight on running with various harnesses — they influence both cost and results.

Conclusion[](https://quesma.com/blog/baba-is-aug-2026/#conclusion)

Progress is fast - with more players joining the frontier game, and open-weight models reaching frontier-level results at a much lower price. And I am looking forward to seeing more models in the coming months. And to the first local model solving the Lake.

As my colleague Piotr Grabowski noticed:

“Second-tier labs caught up with the frontier” matches my experience: last few days I’ve been using Grok 4.6 or GLM-5.3 for easy/medium difficulty tasks and couldn’t see major difference between them and Opus 5/GPT-5.6 Sol (for harder tasks yes, there’s still gap).

And what is your experience?

Previous

HN Claude Code pricing: same tokens, same model, up to 40x the priceClaude Code buyer’s guide, August 2026: what Max, Team, and Enterprise cost, how to get the best deal, and how Uber and Shopify cap the spend.

Related

Kimi K3 is Open, Opus 5 is Good, DeepSeek V4 Flash is Cheap: LLMs on Baba Is YouWe evaluate July 2026 fresh releases Kimi K3, Claude Opus 5, Grok 4.5, Gemini 3.6 Flash, and DeepSeek V4 Flash 0731 on Baba Is Bench, an LLM agent benchmark based on the puzzle game Baba Is You, comparing pass rate, speed, and cost with Claude Fable 5 and GPT-5.6.

Related

Baba Is Solved by Fable 5 and GPT-5.6 Sol, but at what cost?We ported the puzzle game Baba Is You to the Harbor framework, and benchmarked current models, including Claude, GPT, Gemini, GLM, and DeepSeek. A human Twitch streamer is 4x faster than Claude Fable 5.