EngineeringHandling an outage
Process description- Trigger
- Datadog reports that a service is down
- Outcome
- The outage has a clear lead, customers know what is going on and the lessons are written down.
- Tools
- SlackPagerDutySentryStatuspageGitHubIntercomNotion
Steps
- 1Open an incident channelOn-call engineer, tool SlackCan be automated
- 2Page the on-call engineerOn-call engineer, tool PagerDutyCan be automated
- 3Collect errors and recent changesOn-call engineer, tool SentryCan be automated
- 4Set severity and incident leadHead of engineering, tool SlackStays with a person
- 5Update the status pageSupport, tool StatuspageCan be automated
- 6Fix or roll back the changeOn-call engineer, tool GitHubStays with a person
- 7Notify customersSupport, tool IntercomCan be automated
- 8Write the postmortemHead of engineering, tool NotionStays with a person