SWE-rebench Leaderboard

SWE-rebench Leaderboard

❗ SWE-rebench v2LeaderboardPaperDataAboutDiscord

❗ SWE-rebench v2

LeaderboardPaperDataAboutDiscord

15/05/2026

01/07/2026

0123456789101112131415161718192021222324252627282930313233343536

From15/05/2026

To01/07/2026

Click to expand

View

AllAgentModel

LanguageAll

Inspect

Inspect

Potential contamination

External system

Beyond eval range

| # | Model | Resolved Rate (%) | Pass@5 (%) | Cost per Problem ($) | Tokens per Problem | | --- | ----------------------------------------------------- | ----------------- | ---------- | -------------------- | ---------------------- | | 1 | AnthropicFable 5 \[high\]Model | 64.5%± 1.41% | 78.4% | $4.40 | 2,518,30894.9% cached | | 2 | GrokGrok 4.5 \[high\]Model | 63.8%± 0.60% | 77.5% | $1.47 | 2,429,42492.6% cached | | 3 | AnthropicOpus 5 \[high\]Model | 63.4%± 1.35% | 74.8% | $3.47 | 4,322,14395.7% cached | | 4 | Z.aiGLM-5.2 \[high\]Model | 62.9%± 1.19% | 81.1% | $1.40 | 5,524,89292.1% cached | | 5 | OpenAIGPT-5.6 Sol \[medium\]Model | 62.3%± 1.83% | 79.3% | $0.85 | 605,34084.7% cached | | 6 | JunieJunieAgent | 61.8%± 0.54% | 73.9% | $0.81 | 1,684,50185.9% cached | | 7 | AnthropicClaude CodeAgent | 60.4%± 1.03% | 75.7% | $3.39 | 3,341,58193.4% cached | | 8 | OpenAICodexAgent | 58.0%± 1.29% | 73.0% | $1.59 | 2,070,97694.6% cached | | 9 | AnthropicSonnet 5 \[high\]Model | 56.8%± 0.94% | 74.8% | $1.43 | 4,645,61796.4% cached | | 10 | CursorCursorAgent | 51.7%± 0.84% | 65.8% | $0.41 | 1,827,00294.3% cached | | 11 | MinimaxMiniMax M3Model | 47.2%± 1.13% | 69.4% | $0.95 | 13,869,45997.0% cached | | 12 | XiaomiMiMoMiMo V2.5 ProModel | 46.5%± 0.54% | 65.8% | $0.10 | 4,687,98795.7% cached | | 13 | OpenAIGPT-5.6 Luna \[medium\]Model | 43.6%± 1.47% | 59.5% | $0.11 | 395,52285.2% cached | | 14 | DeepSeekDeepSeek-V4 Pro \[high\]Model | 40.2%± 1.29% | 64.0% | $0.15 | 3,955,41491.2% cached | | 15 | QwenQwen3.6-27BModel | 31.2%± 1.68% | 57.7% | $0.62 | 3,078,97076.8% cached | | 16 | QwenQwen3.6-35B-A3BModel | 24.7%± 0.79% | 43.2% | $0.27 | 3,726,06187.5% cached | | 17 | QwenQwen3.5-35B-A3BModel | 17.1%± 1.07% | 36.9% | $0.99 | 5,939,22772.3% cached | | 18 | AnthropicClaude Opus 4.1Model | N/A | N/A | N/A | N/A | | 19 | AnthropicClaude Opus 4.5Model | N/A | N/A | N/A | N/A | | 20 | AnthropicClaude Opus 4.6-highModel | N/A | N/A | N/A | N/A | | 21 | AnthropicClaude Opus 4.7-highModel | N/A | N/A | N/A | N/A | | 22 | AnthropicClaude Opus 4.8-xhighModel | N/A | N/A | N/A | N/A | | 23 | AnthropicClaude Sonnet 3.5Model | N/A | N/A | N/A | N/A | | 24 | AnthropicClaude Sonnet 4Model | N/A | N/A | N/A | N/A | | 25 | AnthropicClaude Sonnet 4.5Model | N/A | N/A | N/A | N/A | | 26 | AnthropicClaude Sonnet 4.6Model | N/A | N/A | N/A | N/A | | 27 | DeepSeekDeepSeek-R1-0528Model | N/A | N/A | N/A | N/A | | 28 | DeepSeekDeepSeek-V3Model | N/A | N/A | N/A | N/A | | 29 | DeepSeekDeepSeek-V3-0324Model | N/A | N/A | N/A | N/A | | 30 | DeepSeekDeepSeek-V3-0324Model | N/A | N/A | N/A | N/A | | 31 | DeepSeekDeepSeek-V3.1Model | N/A | N/A | N/A | N/A | | 32 | DeepSeekDeepSeek-V3.2Model | N/A | N/A | N/A | N/A | | 33 | DeepSeekDeepSeek-V4 Flash \[high\]Model | N/A | N/A | N/A | N/A | | 34 | MistralDevstral-2-123B-Instruct-2512Model | N/A | N/A | N/A | N/A | | 35 | MistralDevstral-Small-2-24B-Instruct-2512Model | N/A | N/A | N/A | N/A | | 36 | MistralDevstral-Small-2505Model | N/A | N/A | N/A | N/A | | 37 | GeminiGemini 3 Flash PreviewModel | N/A | N/A | N/A | N/A | | 38 | GeminiGemini 3 Pro PreviewModel | N/A | N/A | N/A | N/A | | 39 | GeminiGemini 3.1 Pro PreviewModel | N/A | N/A | N/A | N/A | | 40 | GeminiGemini 3.5 FlashModel | N/A | N/A | N/A | N/A | | 41 | Geminigemini-2.0-flashModel | N/A | N/A | N/A | N/A | | 42 | Geminigemini-2.0-flashModel | N/A | N/A | N/A | N/A | | 43 | Geminigemini-2.5-flashModel | N/A | N/A | N/A | N/A | | 44 | Geminigemini-2.5-flash-preview-05-20 no-thinkingModel | N/A | N/A | N/A | N/A | | 45 | Geminigemini-2.5-flash-preview-05-20 no-thinkingModel | N/A | N/A | N/A | N/A | | 46 | Geminigemini-2.5-proModel | N/A | N/A | N/A | N/A | | 47 | GeminiGemma 4 31BModel | N/A | N/A | N/A | N/A | | 48 | Geminigemma-3-27b-itModel | N/A | N/A | N/A | N/A | | 49 | Z.aiGLM-4.5Model | N/A | N/A | N/A | N/A | | 50 | Z.aiGLM-4.5 AirModel | N/A | N/A | N/A | N/A | | 51 | Z.aiGLM-4.6Model | N/A | N/A | N/A | N/A | | 52 | Z.aiGLM-4.7Model | N/A | N/A | N/A | N/A | | 53 | Z.aiGLM-4.7 FlashModel | N/A | N/A | N/A | N/A | | 54 | Z.aiGLM-5Model | N/A | N/A | N/A | N/A | | 55 | Z.aiGLM-5.1Model | N/A | N/A | N/A | N/A | | 56 | Z.aiGLM-5.1Model | N/A | N/A | N/A | N/A | | 57 | OpenAIgpt-4.1-2025-04-14Model | N/A | N/A | N/A | N/A | | 58 | OpenAIgpt-4.1-2025-04-14Model | N/A | N/A | N/A | N/A | | 59 | OpenAIgpt-4.1-mini-2025-04-14Model | N/A | N/A | N/A | N/A | | 60 | OpenAIgpt-4.1-mini-2025-04-14Model | N/A | N/A | N/A | N/A | | 61 | OpenAIgpt-4.1-nano-2025-04-14Model | N/A | N/A | N/A | N/A | | 62 | OpenAIgpt-5-2025-08-07-highModel | N/A | N/A | N/A | N/A | | 63 | OpenAIgpt-5-2025-08-07-mediumModel | N/A | N/A | N/A | N/A | | 64 | OpenAIgpt-5-2025-08-07-minimalModel | N/A | N/A | N/A | N/A | | 65 | OpenAIgpt-5-codexModel | N/A | N/A | N/A | N/A | | 66 | OpenAIgpt-5-mini-2025-08-07-highModel | N/A | N/A | N/A | N/A | | 67 | OpenAIgpt-5-mini-2025-08-07-mediumModel | N/A | N/A | N/A | N/A | | 68 | OpenAIgpt-5.1-codexModel | N/A | N/A | N/A | N/A | | 69 | OpenAIgpt-5.1-codex-maxModel | N/A | N/A | N/A | N/A | | 70 | OpenAIgpt-5.2-2025-12-11-mediumModel | N/A | N/A | N/A | N/A | | 71 | OpenAIgpt-5.2-2025-12-11-xhighModel | N/A | N/A | N/A | N/A | | 72 | OpenAIgpt-5.2-codexModel | N/A | N/A | N/A | N/A | | 73 | OpenAIgpt-5.3-codexModel | N/A | N/A | N/A | N/A | | 74 | OpenAIgpt-5.3-codex-xhighModel | N/A | N/A | N/A | N/A | | 75 | OpenAIgpt-5.4-2026-03-05-mediumModel | N/A | N/A | N/A | N/A | | 76 | OpenAIgpt-5.5-2026-04-23-mediumModel | N/A | N/A | N/A | N/A | | 77 | OpenAIgpt-5.5-2026-04-23-xhighModel | N/A | N/A | N/A | N/A | | 78 | OpenAIgpt-oss-120bModel | N/A | N/A | N/A | N/A | | 79 | OpenAIgpt-oss-120b-highModel | N/A | N/A | N/A | N/A | | 80 | OpenAIgpt-oss-20bModel | N/A | N/A | N/A | N/A | | 81 | GrokGrok 4Model | N/A | N/A | N/A | N/A | | 82 | GrokGrok Code Fast 1Model | N/A | N/A | N/A | N/A | | 83 | OpenRouterhorizon-alphaModel | N/A | N/A | N/A | N/A | | 84 | OpenRouterhorizon-betaModel | N/A | N/A | N/A | N/A | | 85 | KimiKimi K2Model | N/A | N/A | N/A | N/A | | 86 | KimiKimi K2 Instruct 0905Model | N/A | N/A | N/A | N/A | | 87 | KimiKimi K2 ThinkingModel | N/A | N/A | N/A | N/A | | 88 | KimiKimi K2.5Model | N/A | N/A | N/A | N/A | | 89 | KimiKimi K2.6Model | N/A | N/A | N/A | N/A | | 90 | MetaLlama-3.3-70B-InstructModel | N/A | N/A | N/A | N/A | | 91 | MetaLlama-4-Maverick-17B-128E-InstructModel | N/A | N/A | N/A | N/A | | 92 | MetaLlama-4-Scout-17B-16E-InstructModel | N/A | N/A | N/A | N/A | | 93 | MinimaxMiniMax M2Model | N/A | N/A | N/A | N/A | | 94 | MinimaxMiniMax M2.1Model | N/A | N/A | N/A | N/A | | 95 | MinimaxMiniMax M2.5Model | N/A | N/A | N/A | N/A | | 96 | MinimaxMiniMax M2.7Model | N/A | N/A | N/A | N/A | | 97 | OpenAIo3-2025-04-16Model | N/A | N/A | N/A | N/A | | 98 | OpenAIo4-mini-2025-04-16Model | N/A | N/A | N/A | N/A | | 99 | QwenQwen2.5-72B-InstructModel | N/A | N/A | N/A | N/A | | 100 | QwenQwen2.5-Coder-32B-InstructModel | N/A | N/A | N/A | N/A | | 101 | QwenQwen3-235B-A22BModel | N/A | N/A | N/A | N/A | | 102 | QwenQwen3-235B-A22B no-thinkingModel | N/A | N/A | N/A | N/A | | 103 | QwenQwen3-235B-A22B thinkingModel | N/A | N/A | N/A | N/A | | 104 | QwenQwen3-235B-A22B-Instruct-2507Model | N/A | N/A | N/A | N/A | | 105 | QwenQwen3-235B-A22B-Thinking-2507Model | N/A | N/A | N/A | N/A | | 106 | QwenQwen3-30B-A3B-Instruct-2507Model | N/A | N/A | N/A | N/A | | 107 | QwenQwen3-30B-A3B-Thinking-2507Model | N/A | N/A | N/A | N/A | | 108 | QwenQwen3-32BModel | N/A | N/A | N/A | N/A | | 109 | QwenQwen3-32B no-thinkingModel | N/A | N/A | N/A | N/A | | 110 | QwenQwen3-32B thinkingModel | N/A | N/A | N/A | N/A | | 111 | QwenQwen3-Coder-30B-A3B-InstructModel | N/A | N/A | N/A | N/A | | 112 | QwenQwen3-Coder-480B-A35B-InstructModel | N/A | N/A | N/A | N/A | | 113 | QwenQwen3-Coder-NextModel | N/A | N/A | N/A | N/A | | 114 | QwenQwen3-Next-80B-A3B-InstructModel | N/A | N/A | N/A | N/A | | 115 | QwenQwen3.5-27BModel | N/A | N/A | N/A | N/A | | 116 | QwenQwen3.5-397B-A17BModel | N/A | N/A | N/A | N/A | | 117 | StepfunStep-3.5-FlashModel | N/A | N/A | N/A | N/A |

News

• \[2026-07-01\]:
* Added new models to the leaderboad: GLM 5.2, DeepSeek-V4 Pro, DeepSeek-V4 Flash, MiMo V2.5 Pro, Qwen3.6-35B-A3B, Qwen3.6-27B and Gemma 4 31B.
• \[2026-06-09\]:
* Added new models to the leaderboad: Gemini 3.5 Flash and MiniMax M3.
• \[2026-05-28\]:
* Added new models to the leaderboad: Claude Opus 4.8.
• \[2026-05-27\]:
* Added new models to the leaderboad: gpt-5.5-2026-04-23-xhigh, gpt-5.5-2026-04-23-medium, gpt-5.4-2026-03-05-medium, Claude Opus 4.7, and Kimi K2.6.
• \[2026-04-19\]:
* Re-run the Junie with Claude Opus 4.6 as the primary model.
• \[2026-04-15\]:
* Added new models to the leaderboad: GLM-5.1, Qwen3.5-27B, Cursor, Gemma 4 31B and MiniMax M2.7.
• \[2026-03-20\]:
* Added new models to the leaderboard: gpt-5.4-2026-03-05-medium, Gemini 3.1 Pro Preview, Claude Sonnet 4.6, Qwen3.5-397B-A17B, gpt-5.3-codex-xhigh, gpt-5.3-codex and Qwen3.5-35B-A3B * Deprecated following models: gpt-5.2-2025-12-11-xhigh, gpt-5.1-codex-max, gpt-5.1-codex, gpt-5-mini-2025-08-07-high, gpt-5-mini-2025-08-07-medium, Qwen3-235B-A22B-Instruct-2507, DeepSeek-R1-0528, Qwen3-Coder-30B-A3B-Instruct, Qwen3-Next-80B-A3B-Instruct and Qwen3-30B-A3B-Instruct-2507.
• \[2026-03-09\]:
* Added reference evaluation for Junie CLI (highlighted in orange). See setup details in Insights.
• \[2026-02-13\]:
* Added new models to the leaderboard: Claude Opus 4.6, GLM-5, MiniMax M2.5, Codex, Qwen3-Coder-Next, GLM-4.7 Flash, gpt-5.2-codex, GLM-4.7 Flash.
• \[2026-01-14\]:
* Added new models to the leaderboard: gpt-5.2-2025-12-11-xhigh, gpt-5.1-codex, GLM-4.7, gpt-5-mini-2025-08-07-high, gpt-oss-120b-high, Kimi K2 Thinking. * Deprecated following models: gpt-5-2025-08-07-medium, gpt-5-2025-08-07-high, Claude Sonnet 4, Claude Opus 4.1, o3-2025-04-16, gpt-5-codex, GLM-4.5, o4-mini-2025-04-16, gpt-5-2025-08-07-minimal, gpt-4.1-2025-04-14, Qwen3-235B-A22B-Thinking-2507, gpt-4.1-mini-2025-04-14, Qwen3-30B-A3B-Thinking-2507.
• \[2025-12-22\]:
* Added new model to the leaderboard: MiniMax M2.1.
• \[2025-12-17\]:
* Added new models to the leaderboard: gpt-5.1-codex-max, gpt-5.2-2025-12-11-medium, Devstral-2-123B-Instruct-2512, Devstral-Small-2-24B-Instruct-2512, DeepSeek-V3.2. * Added reference evaluation for Claude Code (highlighted in orange). See setup details in Insights. * Deprecated following models: gemini-2.5-pro, gemini-2.5-flash, DeepSeek-V3.1.
• \[2025-12-08\]:
* Added new model to the leaderboard: Gemini 3 Pro Preview.
• \[2025-12-05\]:
* Introduced Cached Tokens column.
• \[2025-11-25\]:
* Added new model to the leaderboard: Claude Opus 4.5.
• \[2025-11-13\]:
* Added new model to the leaderboard: MiniMax M2.
• \[2025-10-28\]:
* Added new model to the leaderboard: GLM-4.6.
• \[2025-10-09\]:
* Added new models to the leaderboard: Claude Sonnet 4.5, gpt-5-codex, Claude Opus 4.1, Qwen3-30B-A3B-Thinking-2507 and Qwen3-30B-A3B-Instruct-2507. * Added a new Insights section providing analysis and key takeaways from recent model and data releases. * Deprecated following models: * Text: Llama-3.3-70B-Instruct, Llama-4-Maverick-17B-128E-Instruct, gemma-3-27b-it and Qwen2.5-72B-Instruct. * Tools: Claude Sonnet 3.5, Kimi K2, gemini-2.0-flash, Qwen3-235B-A22B and Qwen3-32B.
• \[2025-09-17\]:
* Added new models to the leaderboard: Grok 4, Kimi K2 Instruct 0905, DeepSeek-V3.1 and Qwen3-Next-80B-A3B-Instruct.
• \[2025-09-04\]:
* Added new models to the leaderboard: GLM-4.5, GLM-4.5 Air, Grok Code Fast 1, Kimi K2, gpt-5-mini-2025-08-07-medium, gpt-oss-120b and gpt-oss-20b. * Introduced Cost per Problem and Tokens per Problem columns. * Added links to the pull requests within the selected time window. You can review them via the `Inspect` button. * Deprecated following models: * Text: DeepSeek-V3, DeepSeek-V3-0324, Devstral-Small-2505, gemini-2.0-flash, gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14, Llama-4-Scout-17B-16E-Instruct and Qwen2.5-Coder-32B-Instruct. * Tools: horizon-alpha and horizon-beta.
• \[2025-08-12\]: Added new models to the leaderboard: gpt-5-medium-2025-08-07, gpt-5-high-2025-08-07 and gpt-5-minimal-2025-08-07.
• \[2025-08-02\]: Added new models to the leaderboard: Qwen3-Coder-30B-A3B-Instruct, horizon-beta.
• \[2025-07-31\]:
* Added new models to the leaderboard: gemini-2.5-pro, gemini-2.5-flash, o4-mini-2025-04-16, Qwen3-Coder-480B-A35B-Instruct, Qwen3-235B-A22B-Thinking-2507, Qwen3-235B-A22B-Instruct-2507, DeepSeek-R1-0528 and horizon-alpha. * Deprecated models: gemini-2.5-flash-preview-05-20 no-thinking. * Updated demo format: tool calls are now shown as distinct assistant and tool messages.
• \[2025-07-11\]: Released Docker images for all leaderboard problems and published a dedicated HuggingFace dataset containing only the problems used in the leaderboard.
• \[2025-07-10\]: Added models performance chart and evaluations on June data.
• \[2025-06-12\]: Added tool usage support, evaluations on May data and new models: Claude Sonnet 3.5/4 and o3.
• \[2025-05-22\]: Added Devstral-Small-2505 to the leaderboard.
• \[2025-05-21\]: Added new models to the leaderboard: gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14, gemini-2.0-flash and gemini-2.5-flash-preview-05-20.