Codex GPT-6 Sol Performance Tracker
Last updated: Sep 27, 2026
Codex GPT-6 Sol Performance Tracker
We are collecting a new GPT-6 Sol/high baseline from runs beginning September 24, 2026. Degradation detection is paused.
- • Updated daily: Daily benchmarks on a curated subset of SWE-Bench-Pro
- • Collecting baseline: Degradation detection resumes once the new model baseline is established
- • What you see is what you get: We benchmark GPT-6 Sol directly in Codex CLI, with no custom agent harness.
New model — collecting baseline data. Degradation detection paused. View historical performance.
Summary
Status
Degradation Status
Detection is paused while baseline data is collected for the new model.
Collecting baseline data
Baseline
Baseline Pass Rate
Historical average pass rate used as reference for detecting performance changes.
Collecting...
new baseline
Today's Pass Rate
Today's Pass Rate
Percentage of benchmark tasks passed in the most recent day's evaluations.
53 %
50 eval test cases ran
Past 7-day Pass Rate
Past 7-day Pass Rate
Aggregate pass rate over the last 7 days. Provides a more stable measure than daily results.
64 %
300 eval test cases ran
Past 30-day Pass Rate
Past 30-day Pass Rate
Aggregate pass rate over the last 30 days. Best measure of overall sustained performance.
81 %
1,450 eval test cases ran
Daily Trend
Pass rate over time
Toggle 95% CI to view uncertainty around each point.
Daily benchmark pass rates over the past 30 days. Hover over legend items for details on each visual element.
Pass Rate
Daily benchmark pass rate showing the percentage of tasks solved each day.
95% CI
95% confidence interval for each data point. Toggle checkbox to show/hide. Wider intervals indicate more uncertainty (fewer samples).
GPT-6 Sol/high started — 2026-09-24
Loading...
Enable 95% CI checkbox to show confidence intervals
Weekly Trend
Aggregated 7-day pass rate
The same uncertainty toggle applies here for 7-day windows.
Rolling 7-day aggregated pass rates for a smoother trend view with reduced day-to-day noise.
Pass Rate
7-day rolling pass rate aggregating daily results for a smoother trend view.
95% CI
95% confidence interval for each data point. Toggle checkbox to show/hide. Wider intervals indicate more uncertainty (fewer samples).
Loading...
Enable 95% CI checkbox to show confidence intervals
Baseline data is being collected.
Performance deltas will be available once baseline is established.
Change Overview
Performance delta by period
Improvement
Regression
Not Significant
1D Yesterday
\-30.3%
Statistically Significant
\-15% ±11.2% ±11.2% threshold ? With 50 eval test cases, ±11.2% change needed for p < 0.05 50 eval test cases +15%
7D Last Week
\-19.4%
Statistically Significant
\-15% ±3.7% ±3.7% threshold ? With 300 eval test cases, ±3.7% change needed for p < 0.05 300 eval test cases +15%
30D Last Month
\-2.7%
Statistically Significant
\-15% ±1.7% ±1.7% threshold ? With 1,450 eval test cases, ±1.7% change needed for p < 0.05 1,450 eval test cases +15%
Other Metrics
Daily benchmark resource and execution trends
#### Input Tokens
Daily total input usage
Input Tokens
Total input tokens used per daily run.
Loading...
#### Output Tokens
Daily total output usage
Output Tokens
Total output tokens generated by the model in each daily benchmark run.
Loading...
#### Average Runtime
Per-instance runtime by day
Avg Runtime
Average runtime per benchmark instance for each day.
Loading...
#### Tool Calls
Total tool invocations by day
Tool Calls
Total tool invocations made by the agent across the daily run.
Loading...
Methodology
The goal of this tracker is to detect statistically significant degradations in Codex with GPT-6 Sol performance on SWE tasks. We are an independent third party with no affiliation to frontier model providers.
Context: As AI coding assistants become increasingly relied upon for software development, monitoring their performance over time is critical. This tracker helps detect regressions that may affect developer productivity.
We run a daily evaluation of Codex CLI on a curated, contamination-resistant subset of SWE-Bench-Pro. We use the latest available Codex release with GPT-6 Sol. Scheduled evaluations currently use high reasoning effort. Benchmarks run directly in Codex without a custom agent harness, so results reflect what actual users can expect. This allows us to detect degradation related to both model changes and harness changes.
Each daily evaluation attempts N=50 test instances, so daily variability is expected. Pass rates use completed test outcomes; infrastructure failures and cancellations are reported as attempted evaluations but excluded from the statistical denominator. Weekly and monthly results are aggregated for more reliable estimates.
We are collecting a new GPT-6 Sol/high baseline from runs beginning September 24, 2026. The previous gpt-5.6-sol baseline of 83.40% (613 passes among 735 valid trials) belongs to that earlier model and is not used for degradation verdicts during collection. Earlier high and xhigh reasoning runs remain available on the historical performance page.
Pass rates and 95% confidence intervals remain descriptive while the baseline is collected. Degradation detection, significance thresholds, and performance-change comparisons are paused until a baseline for the new model is established.
Get notified when degradation is detected
We'll email you when we detect a statistically significant performance drop.
Email address Subscribe
Thanks for subscribing! Check your email to confirm.
Something went wrong. Please try again.
Interested in promoting your product? Click this button to contact us