A service that normally sees 7,000 to 9,000 requests per second suddenly saw 70,000 to 100,000.
Sit with that number for a second. Ten times the traffic. And here is the part that makes it strange: none of that extra load came from a new feature launch, a marketing push, or a botnet. The amplification came from the client fleet. The very software that was supposed to be politely asking the Copilot Token Service for a token turned around and started shouting at it, all at once.
This is the chapter where the outage stops being a story about GitHub’s datacenter and becomes a story about everyone’s laptop.
GitHub says
Let me stay strict about sourcing, because that is the whole point of 3B.
GitHub says the Copilot Token Service normally sees roughly 7,000 to 9,000 requests per second. During the August 17 incident, it saw roughly 70,000 to 100,000 requests per second. That is about ten times normal.
GitHub attributes the spike to a latent retry bug in VS Code. According to GitHub, delayed responses from an internal endpoint resulted in roughly tenfold request amplification. In plain terms: an endpoint got slow, and because of the bug, slow responses made clients send far more requests than they normally would.
GitHub also says traffic was shifted regionally during the incident. And the recovery timeline is the tell. Most services were back around 16:36 UTC. Actions lagged until about 18:03 UTC. But the Copilot Token Service did not fully recover until about 21:02 UTC, with the incident marked resolved at 21:15 UTC. Copilot took hours longer than almost everything else.
That is the what. A latent client bug, a tenfold spike, and a service that trailed the rest of the recovery by more than four hours. Everything past this point is me reasoning, and I will say so.
The principle
Here is the principle, stated plainly:
Your clients are part of your distributed system. You cannot control how the system behaves by controlling only the servers.
We like to draw a box around “our system” and put it at the datacenter boundary. Inside the box: our services, our load balancers, our databases. Outside the box: users. But that line is a comfortable fiction. Every client running your code is a node in your system that you shipped, that makes decisions on your behalf, and that you do not operate. Its timeout is your timeout. Its retry loop is your retry loop. When millions of copies of it all make the same locally reasonable decision at the same moment, that decision becomes system behavior, and you did not get a vote at runtime. You got your vote when you shipped the client.
What this looks like in your world
You do not need Copilot’s scale to have this problem. You need a fleet of clients and one shared decision.
Think about a mobile app that refreshes on launch. A backend that calls a downstream service with a retry-on-error wrapper. A cron job deployed to a thousand machines that all wake at the top of the hour. In every case, the individual decision is sensible. “The request failed, so I will try again.” “The response is slow, so I will give up and reissue.” No single client is behaving badly. Each one is doing exactly what a careful engineer told it to do.
The trouble is that “each client independently decides to retry a slow request” is a coordination pattern in disguise. Slowness is a shared signal. When the server gets slow, it does not get slow for one client, it gets slow for all of them, at roughly the same time. So they all retry at roughly the same time. That synchronized wave has a name, and it is old: the thundering herd.
And here is the cruel twist specific to this incident. Retries are supposed to help you ride out a blip. But if the retry fires while the original request is still in flight (a timeout that is too aggressive, or a bug that reissues on a delayed response), you have not replaced one request with another. You have doubled it. Do that across a fleet, and a tenfold amplification is not shocking. It is arithmetic.
Why failover did not save Copilot
GitHub shifted traffic regionally during the incident. For a lot of the platform, that is exactly the right move. If Central US is saturated, send the work somewhere with headroom.
But think about what failover actually does. It moves the target. It picks up the destination and sets it down in a healthier region. That works beautifully when the load is a fixed quantity of legitimate demand and the only problem is where it lands.
It does not remove the client-generated load.
If ten times the normal request volume is being generated by client machines all over the world, moving the service to a fresh region can give that traffic more capacity, but it does not reduce the amplification. The same clients continue generating the same excess requests. The herd follows the target. You have not reduced the water, you have only changed which drain it goes down. One plausible interpretation, based on the shape of the incident, is that the other services benefited more directly from traffic rebalancing, easing a resource crunch, while Copilot continued facing amplified client demand that the rebalancing did not shrink.
There is a second, quieter reason, and it is the one I want you to remember longest. A server-side fix does not reach a deployed client. The bug was in VS Code, running on an enormous number of machines that GitHub does not control and cannot restart. Even once the root cause is understood, the fixed client has to be built, released, and then actually installed by users on their own schedule. Old client versions linger in an ecosystem for a very long time. So the pressure does not just stop when someone finds the bug. It can persist while the affected client population continues running the buggy behavior. We don’t know the exact mechanism GitHub used to bring the numbers down (whether it was throttling, an endpoint fix that unstuck the delayed responses, or the fleet churning to newer clients). The report does not say. But the shape of a long tail is consistent with load you cannot switch off from the server side.
Let’s experiment
I want to build the smallest thing that makes this visible, so you can see the amplification emerge rather than take my word for it.
Before the code, hold the herd in your head, because every number below is that picture with instruments attached. Each client is one machine on one desk, politely asking for a token on a steady heartbeat. Add them up and that is the calm baseline: one beat, one request. Now the endpoint slows down, all at once, for everyone. The slowness is the shared signal, so the herd reacts together, and the bug turns each patient request into a stack of impatient ones. The firehose is not aimed at the server. It is the clients.
So I built exactly that. It is small enough to run on a laptop, and it ships with this chapter (link at the end of the section). Two pieces:
Fleet of clients (steady heartbeat) -> one token endpoint (latency I fault)
The Fleet is 1,000 identical clients, each issuing one logical request every 500 milliseconds. That is a steady 2,000 requests per second, my stand-in for the calm 7-9K world. One honest caveat up front: the scale is illustrative, a thousand clients standing in for GitHub’s fleet. The amplification factor is the measured payload, and it does not depend on the absolute numbers.
The Endpoint answers in a fixed time. Healthy, it is fast (20 milliseconds). The fault makes it slow, 500 milliseconds, past the client’s timeout. I give it effectively unlimited capacity on purpose, so the only fault is latency and the only thing that can climb is the load the clients manufacture.
The Client carries the bug: its timeout is an aggressive 50 milliseconds, and on a timeout it reissues immediately, without canceling the request still in flight, without backoff, without jitter, without a budget.
Here is the actual output:
Baseline (steady, healthy world): 2000 req/s
Scenario Endpoint Amplify Success
Healthy baseline 2000 1.0x 100%
Buggy client + slow endpoint 20002 10.0x 100%
Same clients, failover 20002 10.0x 100%
Fix: sane timeout 2000 1.0x 100%
Fix: backoff + jitter 8046 4.0x 100%
Fix: retry budget 2000 1.0x 100%
Read it top to bottom. Healthy, the fleet is a calm 2,000 requests per second, one attempt per beat. Fault the endpoint and turn on the bug, and the load reaching it jumps to about 20,000, a clean tenfold, because with a 500 millisecond response and a 50 millisecond timeout, the client can have roughly ten attempts associated with the same logical request before the original response arrives. The rough tenfold amplification follows from the relationship between the 500 millisecond response time and the 50 millisecond timeout.
Notice the success column: 100% the whole way down. Nothing is failing here. The clients are not fighting errors, they are simply generating ten times the necessary load off one slow response. That is the kind of amplification that can overwhelm a capacity-limited service, even when every individual client believes it is doing the right thing.
Now the row I care about most. Fail over to a fresh region, a fresh endpoint with effectively unlimited capacity in this experiment, and the amplification does not move. Still 10.0x. The clients generate the load, and the clients never changed, so moving the server just points the same firehose at a fresh drain. The herd follows the target.
The fix for the retry behavior has to reach the client, and the bottom three rows are it. A sane timeout, longer than a healthy response, and the amplification collapses to 1.0x, because a sane timeout prevents the client from abandoning requests that are still reasonably likely to complete, eliminating the duplicate in-flight attempts that create the amplification. Backoff and jitter alone bend it partway, to about 4x, spacing the retries out and breaking the lockstep. And a retry budget, a hard cap that says retries may add only a small fraction on top of real traffic and then must fail fast, keeps the amplification bounded in this experiment by preventing retries from continuing indefinitely. The retry budget limits duplicate attempts while the original request is still outstanding. In this experiment the endpoint eventually responds, so logical request success stays at 100%.
Everything here is measured from the real run: the attempts reaching the endpoint, the amplification factor, the success rate. The only thing that is illustrative is the fleet’s size, and the ratio does not care about that.
The code is in the season repo on GitHub, under 06-client-retry-storm/. Run go run ./minimal for the bare baseline-versus-storm table, go run . for the full run above, or go run ./tui to fault the endpoint yourself, press one key to fail over and watch the storm ignore it, then add client discipline and watch it collapse. The lesson the numbers make physical: moving the server does not change the amplification. The fix for a client storm has to reach the client.
Next in 3B: Bit By Byte
We have spent six chapters taking a system apart and watching it run out of things: concurrency, connection flow, retry budget, regional capacity, and now the client fleet itself. Next, in Chapter 7, The Fix Is Not the Lesson, we put it back together. What do you actually change so the next surge is survivable instead of fatal, and what is the one habit underneath all of it that outlasts any single outage?
Subscribe so the last chapter shows up with the rest.