When people talk about leadership, they usually picture someone with a title, a team, and a calm corner office. The version I keep running into looks different. It’s a Monday morning, check-in is down across offices, the PECs can’t get patients into chairs, every waiting room is filling up with people who didn’t want to be at the dentist in the first place, and four of us are on a call waiting for someone to say what we do next.
Writing clean code when CI is green and sprint planning went fine is one skill. Holding a room together when production is down, when stakeholders want an update every ten minutes, and when the team is looking at you for direction, that’s a completely different one. You can be good at the first and still fall apart at the second.
I’ve been on both sides of this: the engineer staring at the dashboard hoping someone else takes the wheel, and later the person who was supposed to have the answers. The moments that defined how my teammates saw me weren’t the well-planned features. They were the bad days. Here’s how I try to lead through them.
Have a Playbook Before You Need It
A surprising number of teams don’t have a real “if things go wrong” plan, especially smaller companies and startups. It’s the engineering equivalent of shipping without backups or a rollback path. Everyone assumes someone else has thought about it.
The time to figure out how to respond is not while the system is on fire. Whether it’s an outage, a serious security issue, or a deploy that quietly corrupts data, the playbook is what stops you from improvising under maximum stress.
In practice it doesn’t need to be heavy. A short doc that covers the basics is enough to start:
- Documented incident response steps, even rough ones.
- A known place to communicate, so people aren’t guessing which channel is the source of truth.
- Clear roles during an incident: who’s driving, who’s talking to stakeholders, who’s heads-down on the fix.
- A few decision trees for the failures you can actually imagine: payment provider down, database in a bad state, a release you need to pull.
You won’t predict every problem. What you get instead is a default starting move, so the first ten minutes go into fixing things rather than into deciding who’s even supposed to be doing what.
The first time I sat in an incident with named roles instead of five people all reading the same logs, the difference was obvious. Less talking over each other. Someone actually watching the customer impact while someone else dug into the cause.
Communicate, Even When You Have Nothing New
During an incident, the thing that erodes trust fastest is silence. No update for thirty minutes and people start to assume the worst, invent their own version of events, and ping you individually, which is exactly when you have the least time to answer.
For engineers, communicating through a crisis isn’t about flooding Slack. It’s about a few specific things:
- Acknowledge the problem fast, before you have a fix. “We know check-in is down across offices and we’re on it” buys you more goodwill than a perfect root cause delivered an hour late.
- Update on a rhythm, even when the update is “still investigating, next update in fifteen.” People can plan around a known cadence. They can’t plan around nothing.
- Be clear about what you don’t know. “I don’t know yet, but here’s what we’re checking” is a complete and honest sentence. It beats a confident guess that turns out wrong, because the wrong guess is what everyone repeats upstream.
The engineers I’ve trusted most in a crisis were rarely the fastest typers. They were the ones who kept everyone oriented, so the rest of us could keep our heads down on the actual problem.
Manage Energy for the Long Haul
The first hour of an incident runs on adrenaline. Everyone’s locked in, watching dashboards, throwing out theories. That part takes care of itself. The danger is what comes after: the root cause that hides for days, the cleanup, the technical debt the fix created, the customer fallout. That can stretch for weeks.
If you burn the team out in the first 48 hours, you have no one left for the part that’s actually slow and grinding. So rotate people out. Make someone go to sleep. Push for notes to be written while things are happening, because nobody will remember the timeline accurately three days later.
This is also where deferred work quietly becomes the next incident. The small cleanups you skip “until things calm down” don’t get cheaper with time, they compound, which is the same trap I wrote about in Parkinson’s Law in Software Engineering. The incident isn’t over when service is restored. It’s over when the mess it left behind is dealt with.
When it’s done, run a blameless postmortem. Not a search for who to blame, a search for what the system let happen. The questions worth asking:
- Could we have caught this earlier, and why didn’t we?
- Were our alerts useful, or just noise people had learned to ignore?
- Did we communicate clearly, internally and to customers?
- Did the way we led the response help or get in the way?
That last one is uncomfortable and worth keeping in. The point isn’t to feel bad. It’s to make the next incident a little less chaotic than this one.
A Few Scenarios I’ve Seen
Some specific high-pressure situations, and what leading through them actually looked like.
The outage during the worst possible window
Check-in going down is not an abstract “service degradation.” It’s full waiting rooms, patients who can’t be seen, and offices that fall back to pen and paper if they can at all, all happening in real time while you watch. The instinct is for everyone to dive straight at the root cause. The better move is almost always to stabilize first: roll back, fail over, degrade gracefully, whatever gets offices moving again, and chase the why once patients are unblocked. Calm is a decision here. If the person coordinating is visibly spiraling, the whole call spirals with them.
When you can’t trust your own signals
The worst incidents I’ve been in weren’t the ones with a clear red alert. They were the ones where the alerts had been crying wolf for months, so nobody believed them, or where the test suite was flaky enough that “red” had stopped meaning anything. When you can’t trust your signals, every decision takes twice as long because you’re debating the instruments instead of the fire. I’ve written about how that erodes a team over time in Why I Care About Fast CI. Leading well in the moment is partly paying down that distrust long before the moment arrives.
Losing a key person mid-project
Sometimes the chaos isn’t technical. A senior engineer leaves, or two do, in the middle of something important, and suddenly a chunk of context walks out the door. The leadership work here is unglamorous: redistribute ownership quickly and explicitly, resist the urge to promise the same timeline as if nothing happened, and protect whoever’s left from quietly absorbing two jobs until they burn out too. Pretending the loss didn’t happen is how you turn one departure into three.
Common Failure Modes
Crisis leadership has its own failure modes, and most of them come from trying too hard.
The first is the hero. The person who fixes everything alone, doesn’t delegate, doesn’t write anything down, and becomes the single point of failure for every future incident. It feels like leadership in the moment, but if the system only recovers when one specific person is awake, you don’t have a resilient team, you have a bus-factor of one.
The second is calling everything a fire. If every minor blip gets the full incident treatment, people stop taking the alarm seriously, and the one time it actually matters they’re already numb to it. Reserve the heavy response for when it’s warranted.
The third is performative calm that’s actually avoidance. Staying composed is good. Staying so “calm” that you downplay a real problem and delay the response is just denial wearing a nicer outfit. The goal is steady, not detached.
What Crisis Leadership Actually Is
Crisis leadership in engineering isn’t about heroics. It’s preparation, communication, and composure, plus the discipline to clean up properly afterward. You don’t need a manager title for any of it. You need to be the person others trust to make clear, steady calls when things go wrong, and to still be standing when the slow part starts.
The next high-pressure moment is probably closer than you’d like. The work is to make sure that when it lands, your team already knows who’s driving, where to look, and what to tell everyone waiting, instead of spending the first ten minutes figuring that out.