incident-management
December 15, 2023 · 6 min read
How to Run Incident Reviews Without Blame
A practical guide to conducting blameless postmortems that improve your systems and strengthen your team culture.
December 15, 2023 · 6 min read
A practical guide to conducting blameless postmortems that improve your systems and strengthen your team culture.
The goal of an incident review isn't to find who to blame—it's to find what to fix. Here's how to run reviews that actually improve your systems and team culture.
When people fear punishment, they hide information. Hidden information prevents learning. Without learning, incidents repeat.
The equation is simple:
Build a detailed timeline before gathering people:
## Incident Timeline: API Gateway Outage - 2024-01-15
### Detection
- 14:23 UTC - PagerDuty alert: API error rate > 5%
- 14:25 UTC - Customer reports via support channel
- 14:26 UTC - On-call engineer Alice acknowledges
### Investigation
- 14:27 UTC - Alice checks Grafana dashboards
- 14:30 UTC - Identifies elevated latency on auth service
- 14:33 UTC - Bob joins incident channel
- 14:35 UTC - Database connection pool exhausted identified
### Mitigation
- 14:38 UTC - Decision to restart auth service
- 14:40 UTC - Rolling restart initiated
- 14:45 UTC - Error rate decreasing
- 14:52 UTC - All metrics nominal
### Resolution
- 14:55 UTC - Incident declared resolved
- 15:10 UTC - Customer communication sent
Ask "why" to find contributing factors, not culprits:
Why did the API return errors?
→ The auth service was timing out
Why was the auth service timing out?
→ Database connection pool was exhausted
Why was the connection pool exhausted?
→ A query was holding connections longer than expected
Why was the query slow?
→ A missing database index on a newly popular query pattern
Why was the index missing?
→ The feature that introduced this query was deployed without load testing
Notice: We never ask "Why did Person X do Y?" We ask about systems, processes, and conditions.
Duration: 45-60 minutes max
Attendees:
Agenda:
Set the tone (2 min)
"This is a learning exercise. We're here to improve our systems, not to assign blame. Everyone did the best they could with the information they had."
Timeline review (15 min)
Contributing factors (15 min)
What went well (5 min)
Action items (15 min)
| Instead of... | Say... |
|---|---|
| "Who deployed the bad code?" | "What was the deployment process?" |
| "Why didn't you catch this?" | "What signals were available?" |
| "Someone should have..." | "What would have helped here?" |
Good questions focus on systems and processes:
When someone gets defensive:
"I appreciate how quickly you responded. Given the monitoring we had at the time, the response was appropriate. Let's talk about what monitoring improvements would help next time."
# Incident Postmortem: [Title]
**Date:** 2024-01-15
**Duration:** 29 minutes
**Severity:** SEV-2
**Authors:** Alice, Bob
**Status:** Complete
## Summary
Brief description of what happened and impact.
## Impact
- 500 API errors for 29 minutes
- ~2,300 affected requests
- 12 customer-reported issues
## Timeline
[Detailed timeline here]
## Root Cause
The database query introduced in PR #1234 performed a full
table scan on the users table during authentication.
## Contributing Factors
1. Missing index on users.last_login_at column
2. No load testing for the new feature
3. Monitoring alert threshold was too high (5% vs 1%)
## What Went Well
- Quick detection (3 minutes from start to alert)
- Clear runbook for auth service restart
- Good cross-team communication
## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| Add index on users.last_login_at | Alice | 2024-01-17 | Done |
| Add load testing to CI pipeline | Bob | 2024-01-30 | In Progress |
| Lower alert threshold to 1% | Charlie | 2024-01-18 | Done |
| Document query review checklist | Alice | 2024-01-25 | Not Started |
## Lessons Learned
- New queries on large tables need explicit review
- Feature flags help with safer rollouts
# Bad
"Improve monitoring"
# Good
"Add alert for database connection pool utilization > 80%
with 5-minute evaluation period. Owner: Alice. Due: Jan 20."
Review action items in team meetings until complete. Incomplete actions from postmortems signal a broken process.
Recognize teams that produce thorough, blameless reviews. Share well-written postmortems as examples.
Leaders must:
When someone makes a mistake and immediately surfaces it, thank them publicly. This reinforces the behavior you want.
Before the review:
During the review:
After the review:
The best incident reviews leave everyone feeling like they learned something, not like they dodged a bullet. That's the culture that builds reliable systems.