Incident Response for Small Teams

The common assumption is that "real" incident response requires a dedicated SRE org, a PagerDuty license, and a thick runbook library. That's wrong. A team of ten shouldn't run incident response with less discipline than a team of three — it should run it with less process, and that should be a deliberate design choice, not an accident.

An On-Call Rotation That Doesn't Burn People Out

The most common mistake small teams make is copying a big company's on-call setup verbatim: 24/7, single-person, weekly rotation. On a three-to-five-engineer team, that means the same person stays on edge for a full week once a month — a pattern that leads to burnout within a few months. A more sustainable model limits on-call hours to business hours and configures after-hours paging for only genuinely critical alerts — payment flow, data-loss risk. Getting paged at 3 a.m. because "disk usage crossed 85%" helps nobody; alerts like that belong in a queue reviewed first thing in the morning, not a buzzing phone.

Severity Levels and a Single Incident Commander

Even a small team benefits from a simple three-tier severity scale: Sev1 (full outage or data-loss risk, immediate response), Sev2 (partial degradation, a subset of users affected), and Sev3 (low impact, can wait for business hours). The critical piece is that every Sev1/Sev2 incident has exactly one person acting as incident commander — that person isn't writing the fix, they're managing coordination: who's doing what, when the next update goes out, when to escalate. The moment two people both slip into "I'm running this" mode, coordination breaks down and the incident drags on longer than it needs to.

The First 15 Minutes: Stabilize, Communicate, Don't Debug Blind

The right move in the first 15 minutes after an incident is reported isn't debugging — it's stabilizing the situation and communicating. The practical order looks like this:

1. Confirm actual impact (are users really affected, or is this a false alarm?)
2. Assign a severity level and name an incident commander
3. Open a status channel (a Slack channel, a status page) and post "we're investigating"
4. Roll back if possible (revert the last deploy, flip off a feature flag) — before you've found the root cause
5. Move to root-cause analysis only after things are stable

Step four is the one people skip most often. Engineers tend to reach for debugging on reflex, wanting to understand why before doing anything else — but rolling back the last deploy is usually a much faster path to stability than a root-cause investigation. As we covered in our piece on retries, backoff, and circuit breakers, a system with its own self-protective mechanisms makes this first 15 minutes far less stressful to begin with.

Google's incident management guide recommends a similar ordering: contain the impact first, understand it second. The most practical way to apply that principle on a small team is keeping "roll back" available as a one-command option at all times — which means every deploy needs to be revertible with a single command. If your rollback mechanism requires several manual steps, it tends to fail exactly when you need it most: mid-crisis.

Blameless Postmortems: the Part That Actually Works

Google's SRE book, in its postmortem culture chapter, makes the point that a postmortem is only truly blameless if it doesn't accuse any individual or team of acting in bad faith — everyone involved is assumed to have done the right thing with the information they had at the time. This isn't just a politeness rule; once a blame culture takes hold, people start hiding mistakes, and hidden mistakes repeat. Removing blame is the only real way to fix systemic issues, because it's what lets someone honestly say "I skipped that step."

For a postmortem to actually work, three things need to hold: the timeline has to reflect what actually happened, not assumptions about what happened; at least one action item has to genuinely land in a sprint (action items written into a postmortem and never done make the whole exercise pointless); and the postmortem needs to be written within 48 hours of the incident, while memory is still fresh. As incident.io's postmortem best-practices piece points out, no matter how good a postmortem template is, if its action items don't have a real owner and a real due date, the document turns into an archive file — a postmortem with no follow-through is only marginally better than no postmortem at all.

It's tempting on a small team to skip the postmortem meeting entirely, since it feels like "we all already know what happened." But that same smallness is exactly what makes the meeting worth holding: on a three- or four-person team, everyone spending 30 minutes in the same room can close out in a single sitting what would take a large company weeks of back-and-forth. Skipping that meeting saves time in the short run, but it raises the odds the same failure repeats three months later.

Alert Hygiene: Fewer Alerts, but the Right Ones

Alert fatigue is the most common reason real incidents get lost in noise. If an engineer gets 20 low-priority pings a day, their ability to tell whether ping number 21 is actually critical erodes fast. The practical rule: every alert should have a clear answer to one of two questions — "does a human need to act on this right now?" or "should we delete this alert or change its threshold?" Everything in the gray zone in between — informational but not actionable — belongs on a dashboard, not on a buzzing phone.

We covered the logs/metrics/traces distinction in more depth in our Observability 101 piece; good alert hygiene starts with being clear about which of those three data types warrants a real-time notification and which one belongs to a morning review instead.

Small Team, Light — but Real — Process

The takeaway here isn't "small teams shouldn't build process" — it's the opposite: build process, just without carrying the weight of a large company's version of it. Check out our zero-downtime deployments piece for deployment practices that reduce incident count in the first place; the best incident response is the incident that never happens. Our database backups and disaster recovery guide also covers the prep work that determines how your first 15 minutes go in the worst Sev1 scenario of all: actual data loss.

Incident Runbook Template

# Incident: <short title>
Severity: Sev1 / Sev2 / Sev3
Incident commander: <name>
Started: <time>

## Status
- Impact: <who/what is affected>
- Current hypothesis: <if any>

## Timeline
- HH:MM - <what happened>

## Actions
- [ ] Status channel opened
- [ ] Impact confirmed
- [ ] Stabilization attempted (rollback/flag flip)
- [ ] Root cause found
- [ ] Permanent fix applied
- [ ] Postmortem scheduled (within 48 hours)

Frequently Asked Questions

Should a three-person team run 24/7 on-call?

Usually not. Limiting on-call to business hours and paging only for genuinely critical alerts after hours protects the team while keeping focus on real emergencies.

Can the incident commander also be the person fixing the issue?

Sometimes unavoidable on small teams, but separating the two roles is the ideal. When the same person coordinates and debugs, one of the two — usually communication — suffers.

Should every incident get a postmortem?

Yes for Sev1 and Sev2. A short note is usually enough for Sev3; writing a full postmortem for every minor incident adds unnecessary process weight and burns the team out.

How often should we review alert thresholds?

Reviewing the relevant alerts after every postmortem is a good habit; sweeping the entire alert list every quarter and asking "is this still needed" prevents alert fatigue from quietly accumulating.