GitHub blames 8-hour outage on autoscaling fail and VS Code retry storm
GitHub has published its account of this week's nearly eight-hour outage, tracing the developer pain to saturated load balancers, a faulty autoscaling policy, and a "latent retry bug in Visual Studio Code."
According to GitHub, problems began at 1328 UTC on August 17 and weren't fully resolved until 2115 UTC – a 7-hour, 47-minute incident that produced elevated errors across Issues, Pull Requests, APIs, Actions, and Copilot.
The immediate cause was network saturation on load balancers in the company's Central US facility, triggered when an Istio sidecar reached its concurrency limit.
REG AD
Surely autoscaling would add capacity as those limits were reached? Alas, no. A misconfigured policy monitored the host service but not the sidecar's concurrency limit, allowing a cascading failure to develop. "The problem," according to GitHub, "was worsened by optimistic retry logic which overloaded internal load balancers."
REG AD
Engineers mitigated the problem by temporarily reducing gateway retries through a code change and configuring the load balancers to reject inbound Copilot Token Service requests with HTTP 403 responses.
Ah yes, Copilot. GitHub explained: "Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service."
Most services recovered by 1636 UTC and Actions by 1803 UTC, but the Copilot Token Service took until 2102 UTC.
"Complicating factors that impeded recovery included a number of scraping attacks on codeload endpoints," GitHub added.
The Microsoft tentacle says it will correct the autoscaling policies, review retry limits, audit Istio concurrency settings, and address the VS Code behavior "that amplified Copilot token traffic."
This latest incident could be the tipping point that sends some developers scurrying for an alternative. CloudBees CEO Moritz Plassnig noted in a LinkedIn post that "Cursor, OpenAI and several smaller startups are already building competitive solutions."
MORE CONTEXT
- Holy git! Microsoft code-sharing site suffers downtime, despite move to Azure
- GitHub has Issues as repo downloads hit 50% error rate
- Latest GitHub outage squeezes Actions, Pages to death
- GitHub Actions outage told devs 'your account is suspended'
-
AI and ml Google buys crashed airline Spirit’s data at auction, because AI
-
SAAS GitHub blames 8-hour outage on autoscaling fail and VS Code retry storm
-
security Expired credit cards revived by researchers to make unauthorized payments
-
ofbeat NASA estimates the size of the hole SpaceX made in the moon
-
virtualization Microsoft ends one of the last ways to buy VMware without big bundles
-
AI and Ml Excel's Copilot function is headed for the Recycle Bin - DEVOPS Go updates may delight diehard gophers but displease AI overlords v 1.27 expands generics to support methods
- EDGE AND IOT Waymo has designed a robocar chip to stay ahead of Tesla 5 nm ML accelerators promise 1,000+ TOPS, ultra-low latency
- SYSTEMS AMD inches closer to its goal of making AI suck less ... energy House of Zen claims latest systems already 4x more efficient than two years ago
- Google pits Marvell against Broadcom as it chases AI crown And Marvell just offered the Chocolate Factory a $12.2B stake to sweeten the deal
- SYSTEMS Cerebras CS-4 rack systems juice chips for every last drop of AI performance Next-gen systems double per-chip performance while cramming 3x as many into a rack
- Security Russians are posing as Signal support to launch phishing attacks PLUS: US takes down Iranian propaganda sites; Marketing company asks 'Why Do We Have Your Information?' And more!
- Security Microsoft patches failed to fix on-prem SharePoint, which is now under zero-day attack PLUS: China upgrades smartphone surveillance tools; Ring eases anti-snooping stance; and more
- Black Hat and DEF CON DEF CON Franklin project enlists hackers to harden critical infrastructure Voting village reports have been so successful, says Jeff Moss, that the whole of DEF CON will now be included
- Security EQT buys majority share in Swiss cybersecurity biz Acronis Went at equivalent of $3.5B+ valuation for entire firm, though portion sold not specified
- Malware Month Ten years since the first corp ransomware, Mikko Hyppönen sees no end in sight On the plus side, infosec's a good bet for a long, stable career
- FOSS smashed one Microsoft monopoly. After 20 years of failure, it's time to smash another Word up
- GNOME can look like Windows – and Flashback can do it without extensions New 'Simple-taskbar' is an option, but there's a simpler, stabler way
- A moment of silence, please, for the final release of Debian on x86-32 New Debian versions hit FOSSland in the form of 13.6 and 12.15
- Baddies caught exploiting extensions bugs with perfect 10 scores on vulnerable Joomla websites Flaws in iCagenda, Balbooa Forms extensions can impact open source CMS that powers a million sites worldwide
- Frame: A new X11 server – implemented directly in assembly Joins yserver, Phoenix, and of course XLibre – and outlier Arcan
- Cinnamon 6.8 will support Wayland – if you want it Next version of Linux Mint’s desktop has both kinds of display server
"GitHub won't be the default solution going forward and we are looking at a much more bifurcated ecosystem (which is good and bad)."
The findings will raise eyebrows among engineers. A misconfiguration and retry storm degraded a critical piece of infrastructure on which many organizations depend, leaving developers unable to work normally for hours.
REG AD
GitHub's reliability problems stretch back well beyond this week, as the company itself acknowledges. Developers have choices, and the pain/gain equation is not looking too favorable for the source shack. As Plassnig noted, alternatives keep cropping up – sometimes at the most awkward moment.
As GitHub staggered, SpaceX-owned Cursor announced an early beta of Origin Code Hosting. ®
[](https://www.facebook.com/sharer.php?u=https%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547) [](https://twitter.com/intent/tweet?url=https%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547) [](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547) [](https://bsky.app/intent/compose?text=GitHub%20blames%208-hour%20outage%20on%20autoscaling%20fail%20and%20VS%20Code%20retry%20storm%0Ahttps%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547) [](https://www.reddit.com/submit?url=https%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547&title=GitHub%20blames%208-hour%20outage%20on%20autoscaling%20fail%20and%20VS%20Code%20retry%20storm) [](https://api.whatsapp.com/send?text=GitHub%20blames%208-hour%20outage%20on%20autoscaling%20fail%20and%20VS%20Code%20retry%20storm%0Ahttps%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547)
REG AD