A service that normally sees 7,000 to 9,000 requests per second suddenly saw 70,000 to 100,000.

Sit with that number for a second. Ten times the traffic. And here is the part that makes it strange: none of that extra load came from a new feature launch, a marketing push, or a botnet. The amplification came from the client fleet. The very software that was supposed to be politely asking the Copilot Token Service for a token turned around and started shouting at it, all at once.

This is the chapter where the outage stops being a story about GitHub’s datacenter and becomes a story about everyone’s laptop.

GitHub says

Let me stay strict about sourcing, because that is the whole point of 3B.

GitHub says the Copilot Token Service normally sees roughly 7,000 to 9,000 requests per second. During the August 17 incident, it saw roughly 70,000 to 100,000 requests per second. That is about ten times normal.

GitHub attributes the spike to a latent retry bug in VS Code. According to GitHub, delayed responses from an internal endpoint resulted in roughly tenfold request amplification. In plain terms: an endpoint got slow, and because of the bug, slow responses made clients send far more requests than they normally would.

GitHub also says traffic was shifted regionally during the incident. And the recovery timeline is the tell. Most services were back around 16:36 UTC. Actions lagged until about 18:03 UTC. But the Copilot Token Service did not fully recover until about 21:02 UTC, with the incident marked resolved at 21:15 UTC. Copilot took hours longer than almost everything else.

That is the what. A latent client bug, a tenfold spike, and a service that trailed the rest of the recovery by more than four hours. Everything past this point is me reasoning, and I will say so.

The principle

Here is the principle, stated plainly:

Your clients are part of your distributed system. You cannot control how the system behaves by controlling only the servers.

We like to draw a box around “our system” and put it at the datacenter boundary. Inside the box: our services, our load balancers, our databases. Outside the box: users. But that line is a comfortable fiction. Every client running your code is a node in your system that you shipped, that makes decisions on your behalf, and that you do not operate. Its timeout is your timeout. Its retry loop is your retry loop. When millions of copies of it all make the same locally reasonable decision at the same moment, that decision becomes system behavior, and you did not get a vote at runtime. You got your vote when you shipped the client.

What this looks like in your world

You do not need Copilot’s scale to have this problem. You need a fleet of clients and one shared decision.

Think about a mobile app that refreshes on launch. A backend that calls a downstream service with a retry-on-error wrapper. A cron job deployed to a thousand machines that all wake at the top of the hour. In every case, the individual decision is sensible. “The request failed, so I will try again.” “The response is slow, so I will give up and reissue.” No single client is behaving badly. Each one is doing exactly what a careful engineer told it to do.

The trouble is that “each client independently decides to retry a slow request” is a coordination pattern in disguise. Slowness is a shared signal. When the server gets slow, it does not get slow for one client, it gets slow for all of them, at roughly the same time. So they all retry at roughly the same time. That synchronized wave has a name, and it is old: the thundering herd.

And here is the cruel twist specific to this incident. Retries are supposed to help you ride out a blip. But if the retry fires while the original request is still in flight (a timeout that is too aggressive, or a bug that reissues on a delayed response), you have not replaced one request with another. You have doubled it. Do that across a fleet, and a tenfold amplification is not shocking. It is arithmetic.

Why failover did not save Copilot

GitHub shifted traffic regionally during the incident. For a lot of the platform, that is exactly the right move. If Central US is saturated, send the work somewhere with headroom.

But think about what failover actually does. It moves the target. It picks up the destination and sets it down in a healthier region. That works beautifully when the load is a fixed quantity of legitimate demand and the only problem is where it lands.

It does not remove the client-generated load.

If ten times the normal request volume is being generated by client machines all over the world, moving the service to a fresh region can give that traffic more capacity, but it does not reduce the amplification. The same clients continue generating the same excess requests. The herd follows the target. You have not reduced the water, you have only changed which drain it goes down. One plausible interpretation, based on the shape of the incident, is that the other services benefited more directly from traffic rebalancing, easing a resource crunch, while Copilot continued facing amplified client demand that the rebalancing did not shrink.

There is a second, quieter reason, and it is the one I want you to remember longest. A server-side fix does not reach a deployed client. The bug was in VS Code, running on an enormous number of machines that GitHub does not control and cannot restart. Even once the root cause is understood, the fixed client has to be built, released, and then actually installed by users on their own schedule. Old client versions linger in an ecosystem for a very long time. So the pressure does not just stop when someone finds the bug. It can persist while the affected client population continues running the buggy behavior. We don’t know the exact mechanism GitHub used to bring the numbers down (whether it was throttling, an endpoint fix that unstuck the delayed responses, or the fleet churning to newer clients). The report does not say. But the shape of a long tail is consistent with load you cannot switch off from the server side.

Let’s experiment

I want to build the smallest thing that makes this visible, so you can see the amplification emerge rather than take my word for it.

Before the code, hold the herd in your head, because every number below is that picture with instruments attached. Each client is one machine on one desk, politely asking for a token on a steady heartbeat. Add them up and that is the calm baseline: one beat, one request. Now the endpoint slows down, all at once, for everyone. The slowness is the shared signal, so the herd reacts together, and the bug turns each patient request into a stack of impatient ones. The firehose is not aimed at the server. It is the clients.

So I built exactly that. It is small enough to run on a laptop, and it ships with this chapter (link at the end of the section). Two pieces:

Fleet of clients (steady heartbeat)  ->  one token endpoint (latency I fault)

The Fleet is 1,000 identical clients, each issuing one logical request every 500 milliseconds. That is a steady 2,000 requests per second, my stand-in for the calm 7-9K world. One honest caveat up front: the scale is illustrative, a thousand clients standing in for GitHub’s fleet. The amplification factor is the measured payload, and it does not depend on the absolute numbers.

The Endpoint answers in a fixed time. Healthy, it is fast (20 milliseconds). The fault makes it slow, 500 milliseconds, past the client’s timeout. I give it effectively unlimited capacity on purpose, so the only fault is latency and the only thing that can climb is the load the clients manufacture.

The Client carries the bug: its timeout is an aggressive 50 milliseconds, and on a timeout it reissues immediately, without canceling the request still in flight, without backoff, without jitter, without a budget.

Here is the actual output:

Baseline (steady, healthy world): 2000 req/s

Scenario                        Endpoint   Amplify   Success
 Healthy baseline                2000       1.0x      100%
 Buggy client + slow endpoint    20002      10.0x     100%
 Same clients, failover          20002      10.0x     100%
 Fix: sane timeout               2000       1.0x      100%
 Fix: backoff + jitter           8046       4.0x      100%
 Fix: retry budget               2000       1.0x      100%

Read it top to bottom. Healthy, the fleet is a calm 2,000 requests per second, one attempt per beat. Fault the endpoint and turn on the bug, and the load reaching it jumps to about 20,000, a clean tenfold, because with a 500 millisecond response and a 50 millisecond timeout, the client can have roughly ten attempts associated with the same logical request before the original response arrives. The rough tenfold amplification follows from the relationship between the 500 millisecond response time and the 50 millisecond timeout.

Notice the success column: 100% the whole way down. Nothing is failing here. The clients are not fighting errors, they are simply generating ten times the necessary load off one slow response. That is the kind of amplification that can overwhelm a capacity-limited service, even when every individual client believes it is doing the right thing.

Now the row I care about most. Fail over to a fresh region, a fresh endpoint with effectively unlimited capacity in this experiment, and the amplification does not move. Still 10.0x. The clients generate the load, and the clients never changed, so moving the server just points the same firehose at a fresh drain. The herd follows the target.

The fix for the retry behavior has to reach the client, and the bottom three rows are it. A sane timeout, longer than a healthy response, and the amplification collapses to 1.0x, because a sane timeout prevents the client from abandoning requests that are still reasonably likely to complete, eliminating the duplicate in-flight attempts that create the amplification. Backoff and jitter alone bend it partway, to about 4x, spacing the retries out and breaking the lockstep. And a retry budget, a hard cap that says retries may add only a small fraction on top of real traffic and then must fail fast, keeps the amplification bounded in this experiment by preventing retries from continuing indefinitely. The retry budget limits duplicate attempts while the original request is still outstanding. In this experiment the endpoint eventually responds, so logical request success stays at 100%.

Everything here is measured from the real run: the attempts reaching the endpoint, the amplification factor, the success rate. The only thing that is illustrative is the fleet’s size, and the ratio does not care about that.

The code is in the season repo on GitHub, under 06-client-retry-storm/. Run go run ./minimal for the bare baseline-versus-storm table, go run . for the full run above, or go run ./tui to fault the endpoint yourself, press one key to fail over and watch the storm ignore it, then add client discipline and watch it collapse. The lesson the numbers make physical: moving the server does not change the amplification. The fix for a client storm has to reach the client.

Next in 3B: Bit By Byte

We have spent six chapters taking a system apart and watching it run out of things: concurrency, connection flow, retry budget, regional capacity, and now the client fleet itself. Next, in Chapter 7, The Fix Is Not the Lesson, we put it back together. What do you actually change so the next surge is survivable instead of fatal, and what is the one habit underneath all of it that outlasts any single outage?

Subscribe so the last chapter shows up with the rest.

Picture the surface of GitHub as a wall of switches. Issues. Pull Requests. The REST and GraphQL APIs. Actions. Copilot. SAML and OIDC sign-in. SCIM. Team Sync. On a normal day these feel like separate products built by separate teams, and in most ways they are. They have their own code, their own databases, their own on-call rotations.

On the afternoon of August 17, a lot of those switches went dark at roughly the same moment.

If you were only watching the symptoms, this looked like a coincidence too big to be a coincidence. How does the thing that opens an issue break at the same time as the thing that reviews a pull request break at the same time as the thing that hands Copilot a token? These features do not call each other. They do not share a database. Someone new to the incident might reasonably guess that ten things broke, which would mean ten separate bugs all deciding to show up on the same Monday.

That is almost never what happened. And the more interesting question is not “which service broke.” It is “which shared dependency did all of these features quietly route through, and why did that one thing get to take all of them down at once.”

That question is what this chapter is about.

GitHub says

Let me stay strict about sourcing, because that is the whole point of 3B.

GitHub’s incident report describes a chain, not a single snap. The immediate trigger was network saturation on load balancers in the Central US datacenter after a new traffic peak. An Istio sidecar hit its concurrency limit and did not scale the way it should have. Retry logic that was trying to help added more load instead of less.

Then comes the sentence I want to sit with for this chapter. GitHub says the original failure cascaded until four HAProxy nodes exhausted their flow limits, which degraded the gateway authentication path and caused widespread authentication latency and failures.

That is the hinge. The features that fell over included Issues, Pull Requests, the APIs, Actions, Copilot, SAML and OIDC authentication, SCIM, and Team Sync. That is a wide list. But notice what almost every item on it has in common. Before it can do its own job, it has to get one question answered somewhere along the request path: who are you, and are you allowed to do this?

When the authentication path at the gateway degrades, that question stops getting answered quickly, or stops getting answered at all. And a feature that cannot confirm who you are cannot safely let you do anything, even if its own logic is completely healthy. The pull request service did not forget how to show you a diff. It just never got past the front door.

That is what GitHub documents. Four HAProxy nodes run out of flow capacity, the gateway auth path degrades, and authentication latency and failures spread across the surfaces that depend on it. On a platform like this, that is most of them. (Exactly which surfaces routed through that path, and whether they all shared one route or several that happened to sit behind the same saturated layer, is a detail the writeup does not settle, and one I come back to at the end.)

The principle

Here is the principle, stated plainly:

Independent features can still share a single road. A gateway, an authentication path, a network layer. And that shared road, not the features themselves, governs their reliability.

We usually draw architecture diagrams as boxes with clean lines between them, and we admire how decoupled it all looks. But there is almost always a layer underneath the boxes that every request passes through on its way in. Call it the shared critical path. It is the dependency that many otherwise unrelated features route through, and when it degrades, all of them degrade regardless of their own health.

Authentication is a perfect example, because it is invisible until it is not. Nobody lists “auth” as a feature. Users do not think about it. But it sits on the entry road for Issues and Pull Requests and the API and Copilot alike. Saturate that one road and you have not broken one feature. You have broken the intersection every feature drives through.

The word for how far a single failure spreads is blast radius. The August 17 story is, in large part, a story about blast radius. A resource ran out (HAProxy flow limits), it degraded one shared thing (the gateway auth path), and because that shared thing sat on the critical path of so many features, the blast radius grew until it looked like the whole platform.

Translating this to your world

You do not run GitHub. You probably do run something with a shared road in it, and you may not have named it yet.

Think about the request lifecycle for your own service. Before your business logic runs, what does every request touch? An API gateway. An auth service or a token check. A service mesh sidecar. A shared connection pool to the primary database. A rate limiter. A DNS resolver. Any one of those is a candidate shared critical path, which means any one of them is a candidate for a blast radius far larger than its own importance suggests.

The test I use is simple. For each shared component, ask: if this gets slow, how many of my supposedly independent features get slow with it? If the honest answer is “most of them,” you have found a road, not a box. Its reliability is not its own private concern. It is the ceiling on everyone else’s reliability.

The good news is that this is a known problem with known shapes of answers. Isolation patterns exist precisely to shrink blast radius. Bulkheads keep one workload from draining a pool that another workload needs. Cells and separate pools give different tenants or different features their own copy of the shared thing, so saturating one does not starve the rest. The point of all of them is the same: make the road narrower in scope so that when it floods, the flood is contained.

Let’s experiment

Before the code, hold the intersection in your head, because every number below is that picture with instruments attached. The three endpoints are three destinations across town, each with a perfectly clear street of its own. The one shared road is the auth path they all merge onto before they can get anywhere. And the flood is one destination’s traffic swelling until it fills that shared road, so cars headed for the other two, healthy streets and all, sit stuck at the same intersection.

So I built exactly that. It is small enough to run on a laptop, and it ships with this chapter (link at the end of the section). It is not a reconstruction of GitHub’s topology, which is not public. It is a deliberately small model of the one mechanism at issue: a shared bottleneck turning one workload’s overload into everyone’s outage. Three independent endpoints, one shared auth gate in front of them:

/notes    (200/s)  \
/tasks    (200/s)  ->  AUTH GATE (30 slots, ~1500 auth/s)  ->  handler (5ms)
/profile  (3000/s) /

The endpoints are /notes, /tasks, and /profile: three separate handlers with zero shared logic, each doing a trivially fast 5ms slice of its own work. That work runs only after the auth slot is released, so the handlers never add to the crowding at the gate. Nothing about them is fragile, and nothing connects them.

The auth gate is the one thing they share. It is a semaphore with 30 slots, and every request has to pass it before reaching its handler. Each auth check holds a slot for 20ms, so the gate can clear only about 1,500 auth checks per second in total, across all three endpoints. A request that cannot get a slot within 50ms is turned away: a timeout at the gate, not a fault in any handler behind it. The request never reached one.

The load is open-loop and flat, one generator per endpoint that never backs off. /notes and /tasks are light and innocent, 200 requests per second each. /profile floods at 3,000 per second, which by itself is twice the entire gate’s capacity.

First run, all three share the one gate. Here is the actual output:

Endpoint   Offered  Success  p50 lat  p95 lat
/notes     200      39%      71ms     83ms
/tasks     200      44%      72ms     94ms
/profile   3000     43%      72ms     86ms

Read the success column first. /profile is flooding, so of course it fails most of what it offers. But look at /notes and /tasks. Each is offering a trivial 200 requests per second, each has a handler that is perfect and untouched, and each is succeeding only about 40% of the time, right alongside the flood. Their latency has tripled too, from a healthy 25ms to over 70ms. Nothing about those two endpoints changed. They simply share /profile‘s road, and /profile filled it. The three success rates fall together, in lockstep, because they were never really three independent systems. They are three tenants of one road.

The specific number is not arbitrary. The gate clears about 1,500 of the 3,400 requests offered each second, roughly 44% overall. All three endpoints pour into the same queue for that one gate, so over any window each endpoint’s share of the requests that get through tracks its share of the requests coming in. That is why all three land near the same 44% success rate, instead of the flood crowding the light endpoints out entirely, and why a light, innocent endpoint pays nearly the same rate as the flood it happens to share a gate with. The exact split drifts run to run with scheduling and timing, but the shape holds.

Now give each its own lane. Same load, same flood, same handlers, but each endpoint gets its own gate instead of sharing one:

Endpoint   Offered  Success  p50 lat  p95 lat
/notes     200      100%     25ms     28ms
/tasks     200      100%     25ms     28ms
/profile   3000     16%      71ms     77ms

The picture splits in two. /notes and /tasks snap back to 100% success at 25ms, because their gates were never crowded to begin with. /profile still tanks, because it genuinely offers far more than one lane can clear: each endpoint’s own gate holds a third of the slots (10 of the 30), so it passes only about 500 auth checks a second, and /profile is still offering 3,000. But now that failure is its own. The flood is exactly as big as before. The only thing that changed is how far the damage reaches. That is a bulkhead: not more capacity, just a wall that keeps one workload’s flood from draining everyone else’s road.

One honest note on those tables: unlike the earlier chapters, there is no modeled number here. The offered rates, the success rates, and the latencies are all measured end to end from real goroutines passing through a real semaphore with real auth waits. Each table is a single run, measured over a three-second window after a one-second warmup, with latency recorded only for requests that succeeded. The whole result is the per-endpoint contrast between the two arrangements, and you can reproduce it.

The code is in the season repo on GitHub, under 05-shared-bottleneck/. Run go run ./minimal for the bare shared bottleneck and one plain table, go run . for the full shared-vs-bulkhead run above, or go run ./tui to dial /profile‘s flood yourself and toggle the arrangement live: watch all three success bars fall together on the shared gate, then press one key and watch /notes and /tasks jump back to full while only /profile stays down. The reasoning stands on its own too: a shared dependency sets the ceiling on everyone who routes through it, and the surest way to shrink a blast radius is to stop sharing the thing that carries it.

What we don’t know

I want to be honest about the edges of the evidence, because that honesty is the discipline.

GitHub tells us four HAProxy nodes exhausted their flow limits and that this degraded the gateway auth path. What the writeup does not hand us is the exact topology: how many features shared those specific nodes, whether every affected surface authenticated through the same path or through several that happened to sit behind the same saturated layer, or what the flow limit numbers actually were. My lab reproduces the mechanism, a shared bottleneck spreading a blast radius, but it is my model of the shape, not a recovered copy of GitHub’s internals. When I say “authentication was the shared road,” I am reasoning from the documented symptom list plus a plainly stated cause. I am not claiming to have seen their config.

That gap is fine. Naming it is what keeps the reasoning trustworthy.

Next in 3B: Bit By Byte

We have been treating the traffic as if it just arrived. It did not entirely. Some of it was manufactured by clients that were trying to be helpful. Next, in Chapter 6, The Copilot Retry Storm, we follow the thread of how a latent client retry bug amplified one service’s traffic from its normal 9,000 requests per second toward 100,000, and what a self-inflicted flood teaches you about the difference between load you receive and load you create.

If you want the whole teardown, one question at a time, subscribe and come with me.

The war of the Mahabharat is a family fight. This is the detail that is easy to forget. The way the story is commonly told, Kurukshetra is not two nations at war. It is one family turning its weapons on itself: cousins raised in the same palace, taught by the same teacher, blessed by the same grandfather. Bhishma, the grandfather who loves both sides, fights for the Kauravas because his vows and his loyalty to the throne bind him. Drona, the teacher, aims arrows at the students he trained. Karna and Arjuna, who (as the story goes) are brothers without both of them knowing it, hunt each other across the field.

And here is what is at stake, so the scene lands even if you have never opened the epic: this is not a war anyone truly wanted. It is a war that, despite repeated attempts to prevent it, eventually became impossible to contain. By the end, almost everyone on that field is dead, on both sides. A whole family destroys itself over a conflict that kept escalating.

Because the way I read it, the tragedy of the Mahabharat is not that a conflict happened. Conflicts happen. The tragedy is how it was fought. It was allowed to grow, over years, from the dice game into a total war, until the conflict had reached a point where the cost would be almost unimaginable. At many points along the way, the fight could have been cooled. Krishna even went as a final emissary of peace, but that effort failed. And so the field is remembered for the slaughter, not for the family it used to be.

The principle

On any team worth being on, conflict is not a sign of failure. It is usually a sign that people care about different things and are willing to say so. A team that never disagrees openly is not necessarily a healthy team. Sometimes it is a team where people no longer feel safe telling the truth. So the goal is not to get rid of conflict. It is to fight it in a way that leaves the team more whole afterward, not less.

There are, roughly, two kinds of conflict, and from a distance they can look the same. One kind is about the work: which approach, whose data, what tradeoff. That kind, argued in the open, tends to make decisions better. The other kind is about ego and position: who is right, who won, who feels threatened. That kind, left alone, slowly turns colleagues into opponents. And once a fight becomes personal, almost no outcome feels good enough, because the fight is no longer about the thing you were actually deciding.

The way I read Kurukshetra, the whole disaster is the story of the first kind of conflict (who should rule, an honest and real question) being allowed to curdle into the second (who must be destroyed). The escalation is the villain here, not the disagreement.

Think of a house fire. A small flame in a pan is a normal part of cooking. The problem is when no one deals with it, and it reaches the curtain, then the wall, then the whole kitchen. The flame was never the danger. Leaving it to spread was. Team conflict works the same way.

The translation

Early in my career, I was leading a small team building an IVR SDK, and two of my most senior engineers could not agree on the architecture. One wanted a single, traditional build. The other wanted to break it into smaller services. At first this was exactly the good kind of fight: two capable people, two real philosophies, argued in the open. I let it run, because it was making the design better.

Then it shifted. The comments in code reviews got sharper. The disagreement stopped being about the design and started being about who was the better engineer. The room went tense. Standups got quiet in the wrong way. That was the moment that mattered, and I almost missed it, because I was still watching the technical argument and not the two people having it.

What fixed it was not picking a winner. I met each of them alone to hear the real concern, then put the two designs into a structured debate with clear criteria, and brought in a respected architect from another team as a neutral outside voice. We landed on a hybrid neither had proposed alone. The feature shipped. But the lesson stayed with me: the disagreement was never the problem. The moment it became personal was.

Watch how a team handles conflict and you can often see whether it will heal or fracture.

• Healing teams fight about the work, out loud, and then decide. The disagreement is on the table, everyone gets heard, a call gets made, and people commit to it, including the ones who lost the argument. The conflict is closed.

• Fracturing teams take it underground. The disagreement goes quiet in the room and loud in the side channels: the DMs, the hallway, the after-meeting meeting. Nothing gets decided, so it just piles up. Like the years between the dice game and the war, resentment compounds until it is no longer about the original issue at all.

• The escalation tell. When people start needing to be right more than they need the team to win, you may be watching a Kurukshetra begin. That is often the moment to step in, before positions harden into vows no one can climb down from.

Say you are a new manager and two engineers on your team disagree about how to build something. At first it sounds healthy: one wants the fast approach, the other wants the safe one, and they argue it in standup. Good. But a week later you notice the arguing has gone quiet, and instead they are each quietly building their own version, cc-ing you on messages that are really about the other person. That is the flame reaching the curtain. The disagreement did not go away. It went underground and turned personal. Your job is not to pick the winner. It is to pull the fight back above ground and back onto the work.

Because a leader’s job in team conflict is usually not to prevent it. It is to keep it about the work, keep it in the open, and close it, so it does not ferment into the kind of conflict that ends teams.

The practice

Next time conflict flares on your team, and at some point it will, try three things:

1. Name which kind it is, out loud. Something like: “I think we’re disagreeing about the approach here, not about each other. Let’s keep it there.” Naming it early helps keep a work conflict from sliding into an ego conflict.

2. Bring the underground conversation up. If the real disagreement is happening in DMs and hallways, say so, gently: “It feels like there’s a concern we’re not talking about openly. Let’s put it on the table.” Silence is where resentment compounds.

3. Close the loop. After the argument, make the decision explicit and ask for commitment, including from the people who did not get their way. An unclosed conflict rarely disappears. It waits.

Every team has its Kurukshetra: a moment when what people care about collides. The field is only remembered for the slaughter because no one cooled the fight in time. You can.

Next week: The Battle Within, the opponent you can’t out-argue.

This is part 6 of a 10-part series on the inner game of organizational life, drawn from the Mahabharat. Subscribe to get one episode every Thursday, one battle at a time.

When the great war of the Mahabharat finally comes, one of the strongest men alive looks at it and walks away.

His name is Balarama. In the way the story is often told, he is Krishna’s elder brother, and one of the mightiest warriors of the age. He had trained fighters on both sides: Bhima among the Pandavas, and Duryodhana among the Kauravas. Both sides knew his strength, and both had reason to want him. This is a war that will kill most of the men on that field and end an entire generation. And Balarama decides it is not his to fight. He leaves on a long pilgrimage and returns only near the end, in time to witness the final confrontation between Bhima and Duryodhana.

It is easy to read that as weakness, as running from the hard thing. The way I read it, it is the opposite. Balarama looked at a fight everyone assumed he would join, and made a deliberate choice: not this one. He seems to have understood something simple. A warrior who fights every battle offered to him is not strong. He is just available.

The Pandavas, who had suffered years of exile and humiliation, make a similar choice from the other direction. They do not begin by demanding the whole kingdom back. As it is commonly told, their final offer is five villages. Five. Not because they do not want the fight, but as a last attempt to secure a settlement they could live with, and avoid a war they know will cost far more than it can ever return. It is Duryodhana, the Kaurava prince, who refuses to give up even a needle’s point of land. He is the one who cannot choose his battles. In the story, that inability destroys him and nearly everyone he loves.

The principle

There are three habits that look separate but are really the same thing. All three are what I mean by choosing your battles.

1. Restraint. Not fighting a battle just because it landed in front of you.

2. Prioritization. Spending your limited energy on the few things that truly matter.

3. Saying no. Declining the requests, meetings, and fights that were never yours to take.

We tend to treat these as small logistics. Calendar tips. Productivity tricks. They are not. They are one discipline, and at heart it is a discipline of self-respect. Every yes is a no to something else, even when you can’t see the something else yet. A person who almost never says no is not more generous than the rest of us. In practice, they have handed the ranking of their own time to whoever asked last and loudest.

Duryodhana’s tragedy, the way I read it, is that he could not tell a battle worth fighting from a battle that was merely available. His pride turned every slight into a war. And someone who fights every battle tends to lose the one that actually mattered, because they show up to it already exhausted.

The translation

A few years ago I made this exact mistake, and it cost real money.

We needed reporting for our product, and early on I built it the quick way: pointing our reporting tool straight at the production database. It worked, and I was proud of how little it took. When teammates suggested we invest in a proper, separate data pipeline, I pushed back every time. Too heavy. Not worth it. We were fine.

We were not fine. As we grew, those reporting queries started dragging down the actual application, the thing customers depended on. Infra costs climbed. We burned engineering weeks patching around a problem I would not name out loud.

The honest truth is that I had stopped making a technical call. I was defending my original decision because it was mine. I had turned “keep the simple setup” into a battle, and I kept fighting it long after the cost stopped making sense.

When we finally moved reporting onto its own pipeline, the application steadied and the noise went away. I lost the battle I should have conceded months earlier because I refused to fight the one that actually mattered.

The lesson was not to avoid difficult decisions. It was to stop spending energy defending a decision that no longer served the work.

For a new manager, the inability to choose battles shows up almost everywhere:

• The meeting you attend because you were invited, not because you are needed. Being available is not the same as contributing.

• The disagreement you win that quietly costs you a relationship you will need next quarter. Some battles are more expensive to win than to lose.

• The pet project you can’t put down even after it has stopped mattering, because your ego is now attached to it.

• The yes you gave because saying no felt rude, and now you resent the person you said it to.

Here is a way to picture it. Think of your attention as a small budget you get to spend once, each day. Every “sure, I’ll take that” is money leaving the account. Most new managers never see the balance drop, because saying yes feels free in the moment. The cost shows up later, on the work that actually had your name on it.

Say a peer asks you to join a working group sorting out a process problem in another team. It is interesting, you have opinions, and saying yes feels helpful. But it is not your team’s problem, and it will eat two hours a week for a quarter, time your own people needed. That is a battle that is available, not yours. Handing it back, kindly, is not laziness. It is you defending the budget for the work only you can do.

Choosing battles well comes down to one honest question, asked before you commit: is this mine to fight, and is it worth what it will cost? Many requests fail one test or the other. The steadiest managers I have watched are rarely the ones who do the most. They are the ones who have decided, clearly, what they will not do, and can say so without a long apology.

None of this is about becoming the person who says no to everything and calls it focus. Choosing your battles means you say yes on purpose too, and then actually show up for the ones you chose.

The practice

This week, run everything asked of you through one filter, and then act on it:

1. “Is this mine?” Is this really my battle, or am I fighting it only because it got handed to me and no one else picked it up? If it is not yours, hand it back, gently and early.

2. “Is it worth the cost?” Not just the hours. The relationship you spend, the focus you lose, the thing you now can’t do. Some wins are not worth their price.

3. Say one clean no. Decline one thing you would normally accept out of habit or guilt. You do not owe a long defense. “I can’t take this on and do it well” is a complete sentence. Notice that the sky does not fall.

Balarama walked away from the biggest war of the age. The way I read it, his decision was not weakness. It was a choice about what he was, and was not, willing to take responsibility for. Duryodhana refused to concede even a needle’s point of land, and lost everything. The difference was not strength. It was the ability to choose.

Next week: Every Team Has Its Kurukshetra. Conflict is inevitable; what you do with it isn’t.

This is part 5 of a 10-part series on the inner game of organizational life, drawn from the Mahabharat. Subscribe to get one episode every Thursday, one battle at a time.

Before the great war in the Mahabharat, Krishna travels to the enemy court on one last mission: to stop the war.

The deal is simple. If the Kauravas give the Pandavas just five villages, everyone goes home and no one dies. Krishna walks into the most powerful court in the land and argues for peace with everything he has.

And Krishna knows how unlikely peace is with Duryodhana.

He knows Duryodhana, the Kaurava prince. He knows how the man thinks. So why make the trip at all?

Not simply to change Duryodhana’s mind. Krishna goes so that when the war comes, no one can say the Pandavas started it. He wants every elder and king in that hall to watch peace get offered honestly, and refused. The meeting he appears to be losing is doing exactly what he needs. He walks out having “failed,” and that failure is the win.

Krishna didn’t go to win the argument in the room. He went because he had already decided what the room was for.

Here is the move, and it is simpler than it sounds. He treated the meeting as a place to confirm an outcome, not to create one. He argued for peace with everything he had. But he wasn’t counting on that argument to move Duryodhana. He was making the case for the record, so the whole hall would watch it play out exactly as he needed.

The principle

Meetings rarely decide things. The deciding mostly happens before, in the quiet. The meeting is where it gets confirmed out loud.

Put another way: an important meeting should almost never be the first place a key person encounters the idea, the problem, or the decision. This is easy to forget. As a new manager, you walk into the big meeting hoping to win it live: on your feet, in front of everyone, in the one room where changing a mind is hardest. That feels like leadership. It is actually the hardest way to get a yes.

Think about why a meeting is a bad place to persuade anyone:

• People are being watched. Changing your mind in front of everyone feels like losing face, so people dig in instead.

• People feel rushed. A real concern needs time to sit with. In the room, they either blurt a half-formed version of it or stay quiet and kill your idea later.

• People are surprised. When someone hears your idea for the first time in the room, their first instinct is to defend, not to consider. Surprise triggers defense.

So the real work happens before the meeting, in the quiet. The pre-read someone actually opened. The 1:1 where the key person got to react without an audience. The hallway sentence that let them object where saving face costs nothing. By the time everyone sits down, the big surprises are gone. Whatever disagreement is left is disagreement you already knew about, and the meeting is where you resolve it in the open, not where you discover it for the first time.

Here is a picture that helps: a wedding. The couple decided to marry weeks ago. The ceremony doesn’t create the marriage. It makes it official and lets everyone witness it. No one walks into their own wedding hoping to win an argument about whether to get married. Treat your important meetings the same way. Do the real work first. Let the room make it official.

So the question is never “how do I perform in the room?” It is “what do I need the room to confirm, and have I secured that yet?”

The translation

Early in my career, I was handed my first big project: build an IVR system for an important client. I thought I knew exactly what we were building. I gathered the team, assigned people, and walked into the kickoff ready to go.

Twenty minutes in, it fell apart. What I thought was a simple system, my boss thought was something three times the size. We had understood the project completely differently, and now that gap was sitting on the table in front of everyone, with people already assigned to the wrong work.

The mistake wasn’t the misunderstanding. It was that I’d let the meeting be the first place we checked whether we agreed. I had assumed alignment instead of confirming it.

So I fixed it the slow way. Before the next project, I sat with each stakeholder privately and got explicit agreement on scope, one conversation at a time, long before anyone gathered in a room. After that, kickoffs stopped being where we discovered disagreements. They became where we confirmed we didn’t have any.

“Winning before the meeting” is not a trick, and it is not politics in a bad sense. It is three ordinary things, done ahead of time.

No one is surprised. If a key person is hearing your proposal for the first time in the room, you are already behind. The people who matter should have seen it privately, with room to object where saving face costs nothing. You are not hiding the ball. You are giving them the chance to say no somewhere it is cheap to say.

Say you want to propose a new on-call rotation. Before the meeting, you send it to the two engineers most likely to hate it and ask, “What am I missing?” One of them points out it breaks during holidays. You fix it. Now, in the room, that objection never even comes up, because you already handled it in private.

The hard questions are already answered. You should know the three toughest objections before you walk in, because you asked the people likely to raise them and either fixed the concern or decided where you’ll hold your ground. The important objections should not be new to you.

You know what the room is for. Is it a decision? Just alignment? Cover from your boss so you can move? A place to talk through a disagreement you already know is there? Krishna’s meeting looked like a negotiation, but it was really about proof: proof that peace was offered. If you can’t say in one sentence what you need the room to confirm, you’ll drift, and the loudest person there will name it for you.

Put simply: you are not going in to create the outcome. You are going in to confirm one you already built. Which means most of your prep is not about the meeting at all. It is the conversations that happen before it. The deck is the smallest part.

The practice

Before your next important meeting, do the work that actually decides it:

1. List everyone whose “no” would sink this, and everyone the decision actually affects. These are often the same people: the ones living with the outcome are the ones whose quiet no shows up later. This is not about hunting for powerful people and getting them onside in secret. Talk to each one before the meeting, even a two-line message, and let them react in private, where they can change their mind without an audience.

2. Write the three hardest questions you could be asked. If you don’t have a real answer to one of them, that is not a meeting problem, it is a homework problem. Fix it before, not during.

3. Finish this sentence: “I need this meeting to confirm ___.” If it’s a decision, know who makes it and what they need to hear to say yes. If it’s alignment, know exactly which known disagreement you’re there to settle.

Do this and the meeting itself becomes almost boring, a formality confirming what you already built. That flat, boring feeling is what winning looks like. Krishna’s face never changed in that court, because nothing in that room was still being decided.

Next week: Choosing Your Battles, why the most strategic word in the language is “no.”

This is part 4 of a 10-part series on the inner game of organizational life, drawn from the Mahabharat. Subscribe to get one episode every Thursday, one battle at a time.

Picture a restaurant on its busiest night. Walk into the kitchen and everything looks calm. The chefs are moving at maybe 40% of what they can do. Nobody is sweating, nothing is on fire, the line is not backed up. If you managed only by looking at the kitchen, you would conclude the restaurant has plenty of room to spare.

Now walk out to the front. There are 200 people waiting at reception. There is exactly one host, and the host can seat one party at a time. The line is out the door and around the corner.

The kitchen graph is green. The restaurant is failing.

Nothing about the kitchen is wrong. The measurement is honest. The chefs really are at 40%. But the number you are staring at has almost nothing to do with the thing that is actually limiting the restaurant tonight. The bottleneck is the host, and no dashboard in the kitchen will ever show it to you.

This is the whole idea of this chapter, and it is one of the most useful mental models the August 17 incident hands us:

CPU is not capacity. Capacity is whichever resource runs out first.

The principle, stated plainly

We tend to talk about a service being “at capacity” as if capacity were one number. It is not. A running service depends on a whole set of resources at once. CPU. Memory. Open connections. Network bandwidth. File descriptors. Concurrency slots in a proxy. Slots in a queue. Threads in a pool.

Your effective capacity is not the average of those. It is the minimum. You can do at most as much work as your most-exhausted necessary resource allows, and not one request more. Everything else can be sitting idle at 40% and it will not help you, in the same way that empty tables do not help when there is nobody free to seat people.

So “are we at capacity?” is the wrong question. The better one is: which resource are we about to run out of first, and are we even watching it?

Translating this to your world

You have almost certainly lived the green-graph version of this.

An incident is happening. Users are seeing errors. And you are looking at a CPU chart that is calm, a memory chart that is calm, and you are thinking: the box is fine, so it cannot be us. Meanwhile the thing that is actually saturated is a connection pool, or a downstream rate limit, or the number of requests a proxy will hold at once. It never showed up on the charts you happened to open, so for the first twenty minutes it did not exist to you.

CPU gets this starring role for a boring reason: it is easy to measure and it is on every dashboard by default. Ease of measurement is not the same as relevance to the bottleneck. That gap is where outages hide.

There is a second half to this that is worth slowing down on, because it is where the incident gets its teeth. It has to do with concurrency, and how it grows even when nothing about your traffic changes.

Concurrency, and why it grows when you are not looking

Concurrency is just the number of requests in flight at the same instant. Not requests per second, which counts how fast they arrive. Concurrency counts how many are sitting inside your system right now, still being worked on.

Here is the part that surprises people. Concurrency can climb even when your request rate is perfectly flat.

Think about it with a mental picture. Suppose 1,000 requests arrive every second, and each one normally finishes in 100 milliseconds. At any given instant, roughly 100 of them are in flight. That is your concurrency. Comfortable.

Now something downstream slows down, and each request starts taking 1 second instead of 100 milliseconds. The arrival rate has not changed at all. Still 1,000 per second. But now each request lingers ten times longer, so roughly 1,000 of them are in flight at once. Your concurrency went up 10x, and your traffic did not move an inch.

That relationship has a name. In plain language: concurrency is roughly the arrival rate multiplied by how long each request takes. (People call this Little’s Law.) You do not need the math. You need the intuition: when latency rises, concurrency rises with it, silently, for free, without any new users showing up. Slowness quietly consumes concurrency.

And concurrency is exactly the kind of resource that has a hard ceiling. A proxy will only hold so many requests at once. When it hits that number, it does not run hot on CPU. It just refuses, or queues, or stalls.

GitHub says

Here is what we actually know, kept strictly to the record.

GitHub reported that during the August 17 incident an Istio sidecar reached its concurrency limit. It also reported that the autoscaling policy was configured around the host service rather than the sidecar’s own resource limits.

Read those two sentences together and the restaurant snaps into focus. The autoscaler was watching the kitchen. The bottleneck was the host. The scaling policy was pointed at the application, and the resource that actually ran out was the concurrency ceiling of the proxy sitting beside it.

Notice what this is not. The autoscaler was not broken. As far as the evidence tells us, it did exactly what it was configured to do. It looked at its metric, saw a healthy number, and correctly decided not to add capacity. A rational decision, made on an honest measurement, of the wrong thing. That is a far more unsettling failure than a bug, because the system was working as designed and still walked straight into the wall.

That gives us the question this whole chapter is really about:

Are we scaling on what is easy to measure, or on what actually limits throughput?

What we do not know, and will not pretend to: the exact numbers, the specific thresholds, or why the policy was pointed where it was. GitHub’s report does not say, so neither do we. The shape is what we are after, and the shape is clear enough.

Let’s experiment

The honest way to trust a claim like “the autoscaler watched the wrong thing” is to build the smallest system where it happens on purpose, and watch it fail.

Before the code, one last translation, because every number below is that scene with instruments attached. The kitchen is the service and the chefs at 40% are its CPU. The one host at the door is the proxy with its concurrency limit. The line out the door is requests in flight. And the manager who decides whether to call in more staff, by glancing only at the kitchen, is the autoscaler.

So I built exactly that. It is small enough to run on a laptop, and it ships with this chapter (link at the end of the section). Same Client -> Proxy -> Service as the last chapter, with two changes that make this one’s point:

Client (flat rate)  ->  Proxy (concurrency limit)  ->  Service (getting slower)

The Client is now open-loop: it fires at a fixed arrival rate, 1,000 requests per second, and never slows down no matter how long requests take. That is the whole setup. The arrival rate is held flat all run, so any rise in concurrency comes from latency alone, not from more traffic.

The Proxy enforces a concurrency limit of 100 per replica, and an autoscaler can add replicas to raise that ceiling. The autoscaler reads one metric several times a second and applies the ordinary scaling rule: if the metric is above target, add capacity.

The Service is healthy, and I make it slower on purpose, step by step: 50ms per request, then 100, then 200, then 400. That is the dependency degrading, each order taking longer to plate.

One measurement note before the tables, because a reader who sees Svc CPU 40% will reasonably assume it was measured. It was not. Service CPU is the one modeled number, not a measured one, because real per-request CPU is too noisy to ship a reproducible result. The model is deliberately simple and stated in the code: CPU tracks the real work the service actually completes, so when the proxy caps throughput, CPU falls with it. Everything else, the in-flight counts, the rejections, and the replica counts the autoscaler settles on, comes straight from the run. And the direction is the part that matters: In this experiment, rising latency never raises CPU, so a CPU-watching autoscaler cannot be provoked into scaling, no matter how bad things get.

Two column notes as well. Offered is the average concurrency the load implies, arrival rate times latency. In-flight is the measured peak, which sits a little above that average because the open-loop client fires in small bursts rather than a perfectly smooth trickle.

First run, the autoscaler watches service CPU. Here is the actual output:

Autoscaler watches SERVICE CPU
Latency  Offered  In-flight  Replicas  Svc CPU  Reject
 (svc)   conc     peak       (scaled)  (model)  rate
 50ms    50       60         1         40%      0%
 100ms   100      100        1         37%      8%
 200ms   200      100        1         19%      51%
 400ms   400      100        1         11%      74%

Read it top to bottom. The arrival rate never moved. But the Offered column climbs 50, 100, 200, 400, exactly as Little’s Law says it must. In-flight requests press against the proxy’s limit of 100 and stop there, because the proxy cannot admit the 101st request concurrently. And service CPU, the one number the autoscaler is watching, does not rise. It falls: 40%, 37%, 19%, 11%. The capped service is completing less real work per second while the overflow piles up at the door, so it looks more idle exactly as the outage gets worse. The autoscaler sees a healthy, falling number, holds at one replica, and by the last row is turning away almost three of every four requests.

Green CPU, failing service, and an autoscaler doing its job perfectly on an honest measurement of the wrong thing.

Now change one thing. Point the same autoscaler at proxy concurrency instead of CPU, and run the identical experiment:

Autoscaler watches PROXY CONCURRENCY
Latency  Offered  In-flight  Replicas  Svc CPU  Reject
 (svc)   conc     peak       (scaled)  (model)  rate
 50ms    50       60         1         40%      0%
 100ms   100      110        2         40%      0%
 200ms   200      210        3         40%      0%
 400ms   400      410        5         40%      0%

Same load, same slowdown, same scaling rule. The only thing we changed was what the autoscaler could see. But now the signal tracks the resource that actually saturates, so as concurrency climbs the autoscaler grows from 1 replica to 5, capacity keeps pace with the in-flight count, CPU holds near its 40% baseline, and nothing is rejected. The manager finally added hosts, because they were finally watching the door.

The code is in the season repo on GitHub, under 03-autoscaling-blind-spot/. Run go run ./minimal for the bare blind spot and one plain table, go run . for the full side-by-side run above, or go run ./tui to dial the latency yourself and toggle the autoscaler's signal live: watch CPU-watching do nothing while the reject bar climbs, then press one key to switch to concurrency and watch the replicas jump and the failures vanish. The reasoning stands on its own too: capacity is set by the first resource to run out, concurrency is a resource, and you cannot scale on a signal you never measured.

Next in 3B: Bit By Byte

There is a loose thread I left dangling. When requests start stalling and failing, the most natural thing in the world happens: everyone tries again. Retries are supposed to make systems more reliable. On August 17 they helped turn a constrained proxy into a much larger outage.

Chapter 4, When Retries Become the Outage. How the thing built to save you becomes the fuel, and what you do about it.

If you want the next question in your inbox when it lands, subscribe. That is the whole ask.

The dashboard is green. Picture the moment. An incident channel is on fire. Requests are failing. Customers are complaining. And you, half-panicked, pull up the metrics for the service everyone is blaming.

CPU: 35%. Memory: comfortable. Latency inside the application code: fine. The health check is passing. By every number you have been trained to trust, this service is healthy.

And yet a large fraction of the traffic aimed at it never gets a clean answer.

This is the moment a lot of engineers (me included, more than once) start to doubt their own tools. The graphs say one thing, reality says another, and it is almost always because you are measuring one resource while a different one ran out. In the August 17 GitHub incident, one of the critical limits was not in the application at all. It was in the proxy sitting beside it.

Let me state the principle before I unpack it, because the whole chapter hangs on this one sentence.

The proxy beside your workload is part of your application’s capacity.

If you only remember one thing, remember that. Now let’s earn it.

GitHub says

I want to stay honest about what is documented and what is me reasoning. So, the facts first.

GitHub’s incident report states that the immediate cause was network saturation on load balancers in Central US after a new traffic peak, and that an Istio sidecar reached its concurrency limit. It also says autoscaling did not react correctly, because the scaling policy watched the host service rather than the sidecar’s limits.

That is the primary evidence for this chapter. Two phrases do the heavy lifting: Istio sidecar and concurrency limit. Everything else I write here is either general mechanism or an experiment. I will keep the buckets separate.

We can reason about: what a sidecar actually is

Here is the part that is safe to reason about, because it is how service meshes work in general, not a claim about GitHub’s private setup.

First, two words that get used interchangeably and should not be.

Istio is the control plane. It is the brain that holds the rules. Which service can talk to which, what the timeouts are, how many concurrent requests are allowed, how traffic gets routed and retried. Istio manages the mesh’s configuration and policy and hands it down; it is not the thing sitting in the request path itself.

Envoy is the data plane. It is the proxy that actually sits in the request path and enforces the rules Istio handed it. Every real request flows through Envoy. Istio writes the law, Envoy is the officer on the road.

A useful compression: Istio decides the rules, Envoy handles the traffic.

Now the sidecar pattern. In Kubernetes, a “pod” can hold more than one container. A common Kubernetes sidecar pattern puts the Envoy proxy in the same pod as the application container. They share a network namespace, so from the application’s point of view the proxy is basically localhost.

The consequence is quiet but important: your application usually does not receive requests directly. Inbound traffic hits the Envoy proxy first, then Envoy forwards it to your app. Outbound traffic leaves your app, hits Envoy, then goes to the world.

                 pod
        +-----------------------+
Client -> [ Envoy proxy ] -> [ Your app ]
        +-----------------------+

Why add a whole extra network hop next to every service? Because that proxy gives you routing, connection pooling, mutual TLS, retries, timeouts, and consistent telemetry, all without changing your application code. You get platform-level traffic behavior for free from the app’s perspective.

But “free from the app’s perspective” is exactly the trap.

That proxy is a real process. It has its own memory. Its own connection pool. Its own CPU. And its own concurrency limit: a cap on how many requests it will handle in flight at the same time.

Concurrency is worth defining precisely, because it is not the same as requests per second. Requests per second is a rate: how many arrive each second. Concurrency is how many are simultaneously in flight right now, still waiting for a response. If requests take 100 milliseconds, a proxy at 1,000 requests per second is holding about 100 in flight at once. If something downstream slows and each request now takes 1 second, that same 1,000 requests per second means about 1,000 in flight. The arrival rate did not change. The concurrency went up 10x.

The relationship is simple: concurrency is roughly throughput times latency (concurrency ≈ throughput × latency). Same arrival rate, 10x the latency, roughly 10x the in-flight work. That single fact is why a small slowdown downstream can saturate a proxy that was comfortable a moment ago.

That is the mechanism to hold onto when reading GitHub’s line that an Istio sidecar reached its concurrency limit. Here is the shape of the failure, reasoned generally. A proxy with a concurrency limit of, say, 1,000 is perfectly fine at 100 in flight. Then latency creeps up somewhere downstream. In-flight count climbs: 300, 600, 900, 1,000. Now the proxy is at its limit. Additional requests may be rejected or forced to wait, depending on how that limit is enforced.

And the application behind that proxy? It is still at 35% CPU. It never even saw those requests. It is idle and healthy while the doorway in front of it is jammed shut.

That is what I mean by “the proxy is part of your capacity.” Your capacity is not CPU. It is not memory. It is not replica count. Your effective capacity is constrained by the first necessary resource to reach its limit, and that resource can be any of:

• Application CPU

• Application memory

• Database connections

• Thread pools

• Network connections

• Proxy concurrency

• Queue depth

Application CPU is only one entry on that list. Proxy concurrency is another. On August 17, per GitHub, the sidecar’s concurrency limit was one of the limits the system could not scale past. The durable question to leave with is not “how does Istio work” but “which of these invisible limits is closest to the edge in my system, and am I graphing it?”

We don’t know

I want to be careful here, because this is the exact place where it is tempting to invent an architecture.

I do not know GitHub’s actual concurrency numbers, how their sidecars were deployed, what the limits were set to, or how their pods were laid out. I do not know whether the picture above matches their topology at all. GitHub’s report tells us a sidecar reached a concurrency limit. It does not hand us the diagram, and I am not going to draw one for them.

What I can do is build a tiny system that has the same shape, and watch the behavior happen with my own eyes.

Let’s experiment

Before the code, hold one picture in your head, because every number below is that picture with instruments attached.

Think of a room with a fire-code capacity of fifty, and one doorway with someone counting people through it. Below fifty, everyone walks straight in. At fifty, the next person waits outside, or gets turned away, no matter how empty the room looks to the people already inside. The doorway is the proxy. Fifty is its concurrency limit. The people in the room are requests in flight, and the room itself is your service: it can feel half-empty while a crowd piles up at its one door.

So I built exactly that. It is small enough to run on a laptop, and it ships with this chapter (link at the end of the section). Three pieces:

Client  ->  Proxy (concurrency limit)  ->  Service (healthy, fast)

The Service is deliberately boring and healthy. It does a tiny slice of work per request (20 milliseconds) and returns. Low load. It is not the villain, and that is the whole point.

The Proxy sits in front of it and enforces a hard concurrency limit: at most 50 requests in flight at once. Anything beyond that is rejected at the door. Envoy provides mechanisms for limiting concurrent requests. For the lab, I model the same constraint with a semaphore: a counting gate that allows only 50 requests to be in flight at a time. I am demonstrating the capacity mechanism, not reproducing Envoy’s internals.

The Client ramps up: 10 concurrent requests, then 30, then 60, then 120.

Then I watch what the service experiences and what the client experiences, side by side. Here is the actual output of one run:

Client  Svc CPU  Svc peak  Proxy   Success
 conc   (model)  in-flight reject  rate
────────────────────────────────────────────
 10     7%       10        0       100%
 30     21%      30        0       100%
 60     35%      50        960     83%
 120    35%      50        6720    42%

Two things happen at once, and they are the whole chapter.

Below 50 concurrent, everything is green: the service load climbs with the client, no rejections, every request succeeds.

Push past 50 and the two stories split. The service’s in-flight count pins at 50 and never climbs again, because the proxy will not admit a 51st request: the room is full. Its load, and so its CPU, has a ceiling that the client cannot push past. But the client’s success rate falls off a cliff: 83% at 60 concurrent, 42% at 120. The proxy is turning the overflow away at the door, and the requests aimed at a perfectly healthy service simply never reach it.

One honest note on that table: service CPU is the one modeled number, not a measured one, because real per-request CPU is too noisy to ship a reproducible result. The model is deliberately simple: the service runs at 35% when it is handling its 50-request ceiling, and scales down from there. Everything else, the rejections, the success rate, the in-flight counts, is measured from the run. The ceiling is the part that matters, and the ceiling is real: the proxy caps how many requests ever reach the service.

That is the entire lesson, made physical. Two numbers telling opposite stories about the same system. The service says “I am at 35% and fine.” The client says “I am failing 58% of the time.” Both are true. The truth lives in the proxy, and nobody was looking at it.

The code is in the season repo on GitHub, under 02-proxy-concurrency/. Run go run ./minimal for the bare mechanism and one plain table, go run . for the full instrumented run above, or go run ./tui to turn the load knob yourself and watch the service bar freeze at the limit while the client bar keeps climbing. Flip on --queue and the failures turn into latency instead of errors; raise --service-ms and throughput falls while the CPU ceiling holds, because the service can only ever clear about fifty requests per service-time. That last one is the earlier concurrency ≈ throughput × latency made live: at a fixed limit, faster requests mean more goodput and slower requests mean less, for exactly the same proxy.

Back to the incident

So when GitHub says a sidecar reached its concurrency limit, this is the mechanism worth carrying: a proxy beside the workload has its own hard limits, independent of the application’s CPU, and when it saturates, the application can look perfectly healthy while the requests aimed at it fail.

Green dashboards are not proof of health. They are proof that the thing you graphed is healthy. The proxy beside your application deserves its own graph.

Which raises the obvious next question. If the app was fine and the proxy was the bottleneck, why didn’t autoscaling notice and add more capacity? Because autoscaling can only react to what it is told to watch, and it was watching the wrong thing.

Next in 3B: Bit By Byte

Chapter 3, When CPU Says Everything Is Fine. We follow the scaling signal that missed the real limit.

If you want the rest of this investigation as it lands, one question at a time, subscribe and follow along.

At 13:28 UTC on August 17, 2026, GitHub.com started experiencing elevated errors and latency.

Issues and Pull Requests were affected. The APIs were affected. Actions was affected. Copilot was affected. Authentication was affected. But they did not all fail in the same way, and they did not all recover at the same time.

By the time it was over, the clock read 7 hours and 47 minutes. That is the headline number, and it is the easy one. Anyone can read that number. Anyone can read the list of things that broke. The interesting question, the one that actually teaches you something, is not what broke. It is why it broke in that order, and what ran out first.

That question is what this whole season is about. In this first chapter I just want to lay out the map and teach you how to read it.

GitHub says

Let me be strict about sourcing, because that discipline is the entire point of 3B. Everything here is from GitHub’s own incident writeup and status page. When I say “GitHub says,” I mean it is documented by the people who were there. When I start reasoning, I will label it. When the evidence runs out, I will say “we don’t know.”

And I am going to stop at the symptoms. Not the causes. The causal chain is the rest of the season, and I refuse to hand it to you as one tidy paragraph you will have forgotten by tomorrow.

The incident ran 13:28 to 21:15 UTC: 7 hours and 47 minutes. In that window a lot of GitHub did not work. Issues and Pull Requests. The REST and GraphQL APIs. Actions. Copilot. Sign-in itself: SAML and OIDC, SCIM, Team Sync. At the peak, roughly one in five web and API requests were failing. For archive and raw content downloads, closer to one in two.

Recovery did not arrive all at once, and the staggering is the clue I want you to carry. Most services were back around 16:36 UTC. Actions dragged until about 18:03. And one service, the Copilot Token Service, did not fully recover until 21:02, hours after everything else. Same incident, wildly different recovery times. Sit with that gap. It is a question this season exists to answer.

That is the what, and I am leaving it there on purpose. GitHub’s report does explain the causal chain, and it is a good one. But I am deliberately not compressing it into a paragraph here. Each link in that chain raises a different systems question, and those questions are where the useful lessons live. Fold it all into one tidy summary and you will nod, feel informed, and learn nothing. So I am going to spend the chapters ahead pulling it apart one link at a time.

The lens

Here is the lens I want you to take from this chapter, stated plainly:

A capacity driven outage rarely stays one failure. One constraint creates pressure somewhere else, and that pressure can become the next failure.

Sometimes that constraint is literal exhaustion. Sometimes it is a concurrency limit, a queue boundary, or another imposed ceiling.

Notice the words “capacity driven.” Not every outage works this way. Plenty are caused by a bad deploy, a config mistake, an expired certificate, corrupted state, a routing error, a plain software defect, and for those, “what ran out” is the wrong question entirely. But when an incident is about capacity, it rarely stays a single failure, because an exhausted resource does not fail politely in place. It pushes load somewhere it was not expected, and that somewhere becomes the next thing to give.

So when I read a capacity related incident, I keep two questions running in a loop:

What constraint was hit? And what did that cause to give next? “What ran out?” is the shorthand I actually say in my head, as long as I remember that not every constraint is literal exhaustion.

August 17 mostly reads that way, though not entirely: a couple of its links are not simple exhaustion, and we will get to those honestly when we reach them. You do not have the pieces to trace the chain yet. That is the point of the chapters ahead. For now, practice the reading itself.

How to think about it

When you read an incident, yours or anyone else’s, hold three buckets separately and refuse to let them blur.

1. What we know. What the evidence explicitly establishes. Timestamps, error rates, the named components. No interpretation yet.

2. What we can reason about. The engineering mechanisms that could explain those observations. This is where you get to be clever, but honestly labeled: you are reasoning, not reporting.

3. What we do not know. Where the public evidence stops. Naming the gap is not a weakness. It is the difference between someone who understands a system and someone who is performing understanding.

I want to get comfortable saying “we don’t know,” because otherwise reasoning quietly turns into storytelling.

Most of this chapter has lived in the first bucket on purpose. We build the evidence base before we build any theories on top of it.

A practice for this week

Find one incident report. GitHub’s, or your own team’s last postmortem, or any public one you can get your hands on.

One rule before you start: do not read the root cause section yet. Read the symptoms, then write down what you think the mechanism was, in your own words, before you look. Only afterward compare your hypothesis against what the report actually concluded. The gap between the two is the lesson. It is the difference between observation leading to hypothesis leading to evidence, and simply copying an answer out of a postmortem.

Read it once for the story. Then read it a second time with a pen. Next to every failure, ask: was something exhausted here, and if so, what? Connections? Concurrency? Memory? Queue capacity? Network flows? Regional capacity? And if nothing seems to have been exhausted, ask what other mechanism produced the failure: a bad deploy, a config change, an expired certificate, a logic bug. Do not stop at “the service was slow.” Slow because of what?

If you cannot name the mechanism, that is not a failure of the exercise. That is you finding an edge of the report, a place where the what is documented but the why is not. Mark it. Those edges are exactly where the interesting work starts.

The repo

Everything in 3B ships with runnable code, and Chapter 1 is no exception. In the season repo I keep the August 17 timeline as a small machine-readable file, plus a short worksheet script that walks you through it one event at a time and asks the only question that matters: what ran out here? It is the practice above, automated, and it is the seed the rest of the season’s lab grows from.

Clone it and follow along:

git clone git@github.com:moksheshd/experiments.git
cd experiments/3b-bit-by-byte/season-01-github-august-17/01-reading-an-outage

The repo: github.com/moksheshd/experiments

Next in 3B: Bit By Byte

We start pulling the first thread. GitHub names an Istio sidecar, a proxy that sits right next to the application, and most engineers have never really looked at it. Next, in Chapter 2, The Proxy Beside Your Application, we find out what that thing actually does, why it has a concurrency limit at all, and why watching the wrong signal to scale it can quietly set a fire.

Some of the questions ahead will sound embarrassingly basic. Why do we even need a proxy there? Why didn’t CPU go up? Why retry something that is already overloaded? Ask them anyway. Basic questions asked relentlessly, held strictly to what the evidence proves, are the whole method.

If you want to follow the whole teardown, subscribe and come along.

Krishna spends the entire war of Kurukshetra in the humblest job on the field.

He is not a king in this war. He carries no weapon; he has vowed not to fight. His role is sarathi, charioteer. He holds the reins while Arjuna shoots. On paper, he is staff, not line. He has no command, no soldiers of his own in the fight, no formal authority over anyone.

And he may be the most influential figure on the entire field.

Kings defer to him. The outcome of the war bends around his judgment. The most important text in the tradition is a record of him talking to the man whose chariot he was driving. He moved everything, and he did it from a seat with no title.

The principle

We are taught that influence flows from authority: get the title, get the corner office, and then people will listen. It’s backwards. Authority tends to be granted to people who already have influence. The title is the lagging indicator. The influence came first.

And influence is not the same as persuasion. This is the part almost everyone gets wrong. We think being influential means being good at convincing: better arguments, sharper rhetoric, more compelling slides. But the influential don’t skip persuasion. They earn the right to barely need it. Persuasion sits downstream of understanding: they start from “what matters to this person,” not “how do I convince them,” so by the time they make the case, they’re already speaking to what the other side actually wants.

Krishna influenced everyone because he understood everyone: every warrior’s ego, every king’s fear, every fracture in every alliance. He didn’t out-argue people. He knew them.

The translation

Early in my career, at a contact-center company in late 2012, I was a regular engineer with no mandate to change anything. Back then, putting video into a contact center meant serious hardware: dedicated video gateways, multipoint units, specialized endpoints, and the licensing that rode on top. It was a capital decision, not a feature you just turned on.

Our VP of Marketing kept talking up a new thing called WebRTC: real-time video straight in the browser, no plugins, no special hardware. It was barely real yet; the first cross-browser video calls were only just happening. But something about it hooked me. Nobody assigned it to me. I taught myself the technology on my own time and built a rough proof-of-concept: a live call running between two browser tabs, no plugin installed, no hardware behind it. Then I wrote up how it could plug into what we already sold, without the hardware bill.

I didn’t walk into a room and argue for it. I showed people the thing working, framed in the terms each of them cared about: for marketing, the story they were already excited to tell; for engineering, less latency and server load; for leadership, a way to leapfrog a huge hardware investment. By the time I asked to lead the effort, there was almost nothing to persuade. That project became a differentiator in our sales pitches, and the title I’d been nowhere near, Module Leader, followed. The influence came first. The title just caught up.

Look around your organization at the people who move rooms without a title. The engineer whose opinion the VP quietly checks. The person two levels down whose “I’m not sure about this” can stall a launch. They are not louder or more senior. They have built an architecture of influence, and it rests on three things, none of which is a job level:

• They understand what people want. Before they make a case, they already know what the other person is trying to protect, win, or avoid. So their “pitch” lands as “here’s how this gets you what you need,” not “here’s why I’m right.”

• They are trusted with the truth. Influence compounds on reliability. The person who tells you the uncomfortable thing when it would be easier to flatter you becomes the person you can’t make a decision without.

• They give before they ask. Krishna served, literally held the reins, long before the moment his counsel decided the war. Influence is something you deposit before you withdraw.

Notice what these share: each is something you do before you need it. You understand people before you need their support. You tell the truth before you need their trust. You give before you need their help. That is the architecture: influence built in advance.

The person chasing influence through authority says: once I’m promoted, they’ll have to listen. The person building influence says: let me understand you so well that a title becomes a formality. Only one of those compounds.

The practice

Pick one person whose support you’ll need for something that matters in the next few months. Before you ever make your case:

1. Write down what they actually want: the real thing they’re optimizing for, in their words, not yours. If you’re guessing, you don’t know them well enough yet. Have the conversation whose only goal is to find out.

2. Do one useful thing for them with no ask attached. Share the context they’re missing, connect them to someone, take a small thing off their plate. Deposit before you withdraw.

3. When you finally make your case, frame it in their terms. Not “here’s why I’m right.” Instead: “here’s how this gets you what you’re trying to get.”

You’ll notice you barely have to persuade. That’s the point. Persuasion is the tax you pay for not having done this earlier.

The man with no title and no weapon moved the whole war. The influence came before the authority.

Next time: why the real conversation almost never happens in the meeting itself.

This is part 3 of a 10-part series on the inner game of organizational life, drawn from the Mahabharat. Subscribe to get one episode every Thursday, one battle at a time.

Before the war, both sides go to Krishna. Duryodhana and Arjuna arrive at Krishna’s chamber at nearly the same moment. He is asleep. Duryodhana, proud, takes the seat near Krishna’s head. Arjuna waits humbly at his feet. When Krishna wakes, his eyes fall on Arjuna first, the one at his feet, and he offers each of them a choice: on one side, his entire army, the mighty Narayani Sena. On the other, Krishna himself, alone, who will not lift a single weapon.

Duryodhana can hardly believe his luck. He picks the army. Arjuna picks the unarmed man.

On the surface, Duryodhana had made the shrewder choice. He got the countable thing: thousands of soldiers. Arjuna got one man who had vowed not to fight. But Arjuna had read the battlefield correctly, and Duryodhana had not. He had confused the visible source of power with the actual one.

The principle

Most people treat organizational politics as warfare, a thing you win by fighting harder, pushing more, out-arguing the other side. It isn’t. Politics is terrain. It is the shape of the ground you’re standing on: where the power actually sits, what each person is really trying to protect, and which relationships actually matter when a decision gets made.

You cannot out-fight bad terrain. An army on high ground beats a better army in the valley. And in an organization, the person who has read the terrain will beat the person who simply came prepared to fight. The one who knows that the “decision-maker” defers to someone three doors down, that the loud objection in the meeting is theater and the real veto came by DM an hour earlier.

Duryodhana counted soldiers. Arjuna read the field.

The translation

A few years ago, I pushed to break our main system apart into smaller pieces. When we first entered the market it had been one simple monolith, and that was the right call: it gave us speed, we shipped fast. But two or three years in, the client base had grown and the same design was creaking. Performance was slipping, and the code had turned into a tangle nobody wanted to touch.

I was sure I was right, and I had the data to prove it: the bottlenecks, the graphs, the whole case. I walked into the room expecting the argument to win, because the argument was correct.

It went nowhere. The senior architects and a few of the team looked at my charts, nodded, and quietly did not move. I pushed harder. More data, cleaner slides, the same polite nothing.

What I had misread wasn’t the technology. It was what the people in that room were actually protecting. I thought they were defending the old design. They were defending simplicity: one codebase, one deployment, one place to hold the whole system in your head, and a way of working they had gotten fast at. The new AI coding assistants reinforced that advantage. They could read the whole thing at once and generate code against it easily, and breaking it into services put that simplicity at risk. The decision was never going to be settled by whoever had the best benchmark. It was going to be settled by what those senior people were unwilling to give up.

Once I understood that, I stopped arguing performance. I started answering the thing they actually cared about. I took one small, low-risk piece, the report-generation logic, pulled it out into its own service, and showed it could be built with the same AI tools, deployed on its own, and run without the mess they were bracing for. That one working example did what my benchmarks never could. It didn’t win everyone over, but it earned me enough room to take the same approach to the core logic that mattered most.

I had spent weeks aiming at the wrong target.

Every organization has two maps. There is the org chart, the ranbhoomi as it’s officially drawn, and there is the real map of how things actually get decided. The two are rarely the same. Reading the battlefield means learning the second one.

Ask, before any move:

• Where does power actually sit? Titles tell you where authority is supposed to be. Watch instead for whose name gets invoked, whose small objection stops a project, whose approval people seek before they seek the official one.

• What is each person actually trying to protect? Not the stated goal. The real one. The manager guarding headcount. The peer whose priorities are threatened by your project. Most people make sense once you know what they’re really trying to hold on to.

• Which relationships actually matter? Some people can stop a decision cold. Most can’t, however friendly they seem. Back the wrong one and you’re counting on support that was never really there.

None of this is cynical. It’s just how things actually work. You can insist the org chart is reality all you want. The people making the real decisions won’t notice.

Krishna, unarmed, standing in Arjuna’s chariot, turned out to matter more than an army. Not because he fought, but because he could see the whole field and the people on it, the judgment no number of soldiers can buy. The way I read it, that is why one unarmed man mattered more than Duryodhana’s thousands.

The practice

Draw your organization’s real map on paper. For the decision you care about most right now:

1. Name the official decision-maker. Then ask: who do they check with before they decide? That person may have more practical influence than the org chart suggests.

2. For each key player, write one line: “What do they actually need to protect or win here?” If you can’t answer it, you don’t understand the terrain yet, so go find out before you move.

3. Mark the relationships that can stop this. Who, if they said no, ends it? Those are the ones that matter. The rest is noise.

Keep it private. Redraw it every quarter, because terrain shifts. But once you can see the real map, you’ll stop fighting uphill by accident.

Next week: How the unarmed man in the chariot became the most influential figure in the war.

This is part 2 of a 10-part series on the inner game of organizational life, drawn from the Mahabharat. Subscribe to get one episode every Thursday, one battle at a time.