GitHub blames 8-hour outage on autoscaling fail and VS Code retry storm
GitHub has published its account of this week's nearly eight-hour outage, tracing the developer pain to saturated load balancers, a faulty autoscaling policy, and a "latent retry bug in Visual Studio Code."
According to GitHub, problems began at 1328 UTC on August 17 and weren't fully resolved until 2115 UTC – a 7-hour, 47-minute incident that produced elevated errors across Issues, Pull Requests, APIs, Actions, and Copilot.
The immediate cause was network saturation on load balancers in the company's Central US facility, triggered when an Istio sidecar reached its concurrency limit.
REG AD
Surely autoscaling would add capacity as those limits were reached? Alas, no. A misconfigured policy monitored the host service but not the sidecar's concurrency limit, allowing a cascading failure to develop. "The problem," according to GitHub, "was worsened by optimistic retry logic which overloaded internal load balancers."
REG AD
Engineers mitigated the problem by temporarily reducing gateway retries through a code change and configuring the load balancers to reject inbound Copilot Token Service requests with HTTP 403 responses.
Ah yes, Copilot. GitHub explained: "Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service."
Most services recovered by 1636 UTC and Actions by 1803 UTC, but the Copilot Token Service took until 2102 UTC.
"Complicating factors that impeded recovery included a number of scraping attacks on codeload endpoints," GitHub added.
The Microsoft tentacle says it will correct the autoscaling policies, review retry limits, audit Istio concurrency settings, and address the VS Code behavior "that amplified Copilot token traffic."
This latest incident could be the tipping point that sends some developers scurrying for an alternative. CloudBees CEO Moritz Plassnig noted in a LinkedIn post that "Cursor, OpenAI and several smaller startups are already building competitive solutions."
MORE CONTEXT
- Holy git! Microsoft code-sharing site suffers downtime, despite move to Azure
- GitHub has Issues as repo downloads hit 50% error rate
- Latest GitHub outage squeezes Actions, Pages to death
- GitHub Actions outage told devs 'your account is suspended'
-
AI and ml Google buys crashed airline Spirit’s data at auction, because AI
-
AI and Ml Excel's Copilot function is headed for the Recycle Bin
-
DEVOPS GitHub has Issues as repo downloads hit 50% error rate
-
security Expired credit cards revived by researchers to make unauthorized payments
-
PUBLIC SECTOR Capita handed vital role in next pandemic despite civil service pension failures
-
software Microsoft MVP creates site to remind you of all the brands Redmond replaced - SYSTEMS Cerebras CS-4 rack systems juice chips for every last drop of AI performance Next-gen systems double per-chip performance while cramming 3x as many into a rack
- Security Copilot tricked into telling reseachers how to hack itself How to social engineer an AI's reasoning engine
- AI and ml Payments giant Stripe is about to drop over $7 billion to become a gateway to AI token sales AI gateways look promising as companies struggle with model orchestration
- ai and ml Anthropic says text watermarking scheme relies on inconsequential words 'Shall I compare thee to a summer's afternoon' is the sort of thing this will make, and others look likely to adopt it
- AI and ML DeepSeek's innovative harness treats everything as a plug-in Chinese AI labs keep moving forward while US labs play defense
- Security Russians are posing as Signal support to launch phishing attacks PLUS: US takes down Iranian propaganda sites; Marketing company asks 'Why Do We Have Your Information?' And more!
- Security Microsoft patches failed to fix on-prem SharePoint, which is now under zero-day attack PLUS: China upgrades smartphone surveillance tools; Ring eases anti-snooping stance; and more
- Black Hat and DEF CON DEF CON Franklin project enlists hackers to harden critical infrastructure Voting village reports have been so successful, says Jeff Moss, that the whole of DEF CON will now be included
- Security EQT buys majority share in Swiss cybersecurity biz Acronis Went at equivalent of $3.5B+ valuation for entire firm, though portion sold not specified
- Malware Month Ten years since the first corp ransomware, Mikko Hyppönen sees no end in sight On the plus side, infosec's a good bet for a long, stable career
- FOSS smashed one Microsoft monopoly. After 20 years of failure, it's time to smash another Word up
- GNOME can look like Windows – and Flashback can do it without extensions New 'Simple-taskbar' is an option, but there's a simpler, stabler way
- A moment of silence, please, for the final release of Debian on x86-32 New Debian versions hit FOSSland in the form of 13.6 and 12.15
- Baddies caught exploiting extensions bugs with perfect 10 scores on vulnerable Joomla websites Flaws in iCagenda, Balbooa Forms extensions can impact open source CMS that powers a million sites worldwide
- Frame: A new X11 server – implemented directly in assembly Joins yserver, Phoenix, and of course XLibre – and outlier Arcan
- Cinnamon 6.8 will support Wayland – if you want it Next version of Linux Mint’s desktop has both kinds of display server
"GitHub won't be the default solution going forward and we are looking at a much more bifurcated ecosystem (which is good and bad)."
The findings will raise eyebrows among engineers. A misconfiguration and retry storm degraded a critical piece of infrastructure on which many organizations depend, leaving developers unable to work normally for hours.
REG AD
GitHub's reliability problems stretch back well beyond this week, as the company itself acknowledges. Developers have choices, and the pain/gain equation is not looking too favorable for the source shack. As Plassnig noted, alternatives keep cropping up – sometimes at the most awkward moment.
As GitHub staggered, SpaceX-owned Cursor announced an early beta of Origin Code Hosting. ®
[](https://www.facebook.com/sharer.php?u=https%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547) [](https://twitter.com/intent/tweet?url=https%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547) [](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547) [](https://bsky.app/intent/compose?text=GitHub%20blames%208-hour%20outage%20on%20autoscaling%20fail%20and%20VS%20Code%20retry%20storm%0Ahttps%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547) [](https://www.reddit.com/submit?url=https%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547&title=GitHub%20blames%208-hour%20outage%20on%20autoscaling%20fail%20and%20VS%20Code%20retry%20storm) [](https://api.whatsapp.com/send?text=GitHub%20blames%208-hour%20outage%20on%20autoscaling%20fail%20and%20VS%20Code%20retry%20storm%0Ahttps%3A%2F%2Fwww.theregister.com%2Fsaas%2F2026%2F08%2F19%2Fgithub-blames-8-hour-outage-on-autoscaling-fail-and-vs-code-retry-storm%2F5289547)
REG AD