Postmortems and Jeli Post Incident Reviews
Document and learn from past major incident responses
Postmortems are reports that allow users to learn from major incidents by providing a summary of events that transpired, how the response was handled, and what resolution steps were taken. Postmortems curate a timeline of activity in PagerDuty and Slack, which teams can analyze to determine follow-up actions for similar situations in the future and, ideally, help prevent them altogether.
Jeli Post Incident Reviews are a structured evaluation process conducted after an incident or crisis to analyze what happened, why, and how to prevent recurrence. It's a valuable tool for organizations to learn from mistakes, improve response procedures, and enhance overall resilience in the face of future challenges.
Pricing Plan
The Postmortems feature is available for accounts on our Business, Enterprise for Incident Management and Digital Operations (Legacy) plans. If you are interested in a pricing plan that includes PagerDuty Postmortems, please contact our Sales team for more information.
Create a Postmortem
Required User Permissions
Observers, Limited Users, Users, Admins and Account Owners (or their respective advanced permission base roles) can create, edit and delete postmortems.
- Navigate to Incidents Postmortems catalog and click New Report, or click the New Postmortem Report button on a resolved incident.
-
- If you created a new postmortem from the catalog page: Enter a Report Name and Owner of Review Process. You may optionally enter a Impact Start Time and Impact End Time. Keep in mind that, although multiple people may have been involved in the incident’s response, there can only be a single owner of any one postmortem. Once finished, click Create Report.
- If you created a new postmortem from a resolved incident: Your report details will be auto populated and you will be assigned as the Owner of Review Process. You may choose to edit these details if necessary. Once finished, click Create Report.
- Data Sources: This section allows you to build a timeline of incident log entries and Slack messages related to the incident.
- PagerDuty: To link incidents to this postmortem, either Select incident between impact start and end times from the dropdown, or Find by incident number by entering incident numbers (one at a time) and click Add This Incident.
- Slack: If you’d like to include discussions from public Slack channels, you can do so by clicking the Connect to Slack button. Once you’ve authorized PagerDuty to access your Slack account, you will then be prompted to search and select Channels to include in your report.
Slack Data Source
To include Slack content in a postmortem, a contributor must authorize PagerDuty to access Slack on their behalf, which will let that contributor see Slack content in the Available Data listing. Each PagerDuty user will need to authorize with Slack individually.
If your Slack user account does not have permissions to add apps to your team, you will need to first ask a Slack admin to go to Slack's App Directory and authorize the PagerDuty Postmortems app.
Click Save Data Sources to continue.
- Timeline: All Slack messages and incident log entries from the above data sources will be listed in the Available Data (left) slush bucket. The incident Trigger and Resolve log entries will be listed in the Included in Timeline (right) slush bucket by default.
- To add Slack messages or incident log entries to the timeline: Select the event and click the button, or click the to the left of the event to move it from the left to the right slush bucket. You can add multiple items at once by selecting them while holding the shift key and then using the button to move them to the timeline. You can also manually add freeform entries to your timeline by selecting New Timeline Entry. Any Slack entries that are moved from the Available Data listing into the postmortem's Timeline will be visible to anyone looking at the postmortem.
- To remove an event from the timeline: Select the event and click the button, or click the to the left of the event to move it to the left slush bucket.
Click Save Timeline to proceed.
- Analysis: This section allows you to summarize and reflect on the curated timeline, including incident root cause analysis and potential steps to prevent or mitigate future incidents.
Fill in the following fields based on your analysis:
Field | Description |
---|---|
OVERVIEW | Include a short sentence or two summarizing the contributing factors, timeline summary, and the impact; for example, "On the morning of August 14th, we suffered a 14 minute SEV-1 due to a runaway process on our primary database machine. This slowness caused roughly 3% of reports generated during this time to be completed out of SLA." |
WHAT HAPPENED | Include a short description of what happened, usually based on the timeline. |
RESOLUTION | Include a description what solved the problem. If there was a temporary fix in place, describe that along with the long-term solution. |
ROOT CAUSES | Include a description of any conditions that contributed to the issue. If there were any actions taken that exacerbated the issue, also include them here with the intention of learning from any mistakes made during the resolution process. |
IMPACT | Be specific here. Include numbers such as customers affected, cost to business, etc. |
WHAT WENT WELL? | List anything you think we did well and want to call out. It's okay to not list anything. |
WHAT DIDN'T GO SO WELL? | List anything you think we didn't do very well. The intent is that we should follow up on all points here to improve our processes. |
ACTION ITEMS | Include action items such as: (1) fixes required to prevent the issue in the future, (2) preparedness tasks that could help mitigate a similar incident if it came up again, (3) remaining postmortem steps, such as an internal follow-up email, updating the public status page, etc. |
NEW SECTION | If you would like to add a new section, you may do so here. |
Customized Postmortem Templates
The fields above are from the default Postmortem Template. Some fields may be subject to change based on how your postmortems have been customized.
Each analysis field may be edited with the following options by clicking the to the right:
- Edit Section Title
- Move to Top
- Move Up
- Move Down
- Move to Bottom
- Delete Section
You may also add new sections by clicking Add Section.
- Once you have completed and reviewed your report, click Save & View Report.
Edit a Postmortem
- Navigate to Incidents Postmortems to the right of the postmortem Edit. Or you may select the postmortem’s Name and click Edit Report.
- Edit any details necessary and then click Save & View Report.
Delete a Postmortem
- Navigate to Incidents Postmortems to the right of the postmortem Delete. Or you may select the postmortem’s Name and click Edit Report and then click Delete Report.
- A confirmation dialog will appear. Click Delete to delete your postmortem.
Export a Postmortem
Once you have completed your postmortem report, you can export it as a PDF.
- Navigate to Incidents Postmortems and select your desired postmortem's Name.
- Click the Save as PDF button.
Customize the Postmortem Template
The purpose of the postmortem process is to learn how to improve both system resiliency and the response process. As you learn new insights, you can customize your report and add or remove sections that apply to your team's specific use case.
- Navigate to Incidents Postmortems Settings tab.
-
- To add a new section to your template: Scroll to the default New Section, click the to the right and select Edit Section. You may then enter a new section Title and a Description and then click Save. To continue adding new sections, click the Add Section button at the bottom of the page.
- To edit existing sections: Click the to the right and select Edit Section. Make any edits necessary and then click Save.
- To delete sections: Click the to the right and select Delete. A confirmation dialog will appear. Click Yes, Delete Section to delete the section.
Reset the Postmortem Template
You can also reset your postmortem template to the default sections if needed.
Resetting Templates
Resetting your Report Template to its default will only affect future postmortem reports. This change will not affect existing postmortem reports.
- Navigate to Incidents Postmortems Settings tab.
- Click the Reset Template button at the top of your Report Template. A confirmation dialog will appear. Click Reset Template to reset the template to its default.
Jeli Post Incident Reviews
Users on Enterprise Incident Management or Customer Service Operations pricing plans have the option to create a post-incident review in Jeli.
Pricing Plan
Jeli is available on our Enterprise for Incident Management and Enterprise Customer Service Ops Plans. If you are interested in a pricing plan that includes PagerDuty Postmortems, please contact our Sales team for more information.
Configure Jeli
In order to use Jeli, there are a few required setup steps:
-
In the PagerDuty web app, navigate to Integrations API Access Keys.
-
Click Create New API Key and enter "Jeli API Key." Copy the newly created API key to your clipboard, or save it in a secure location.
-
From the top navigation bar, select Incidents Jeli Post Incident Reviews. When prompted, sign in to Jeli using the option Continue with PagerDuty.
-
In the Jeli web app, select Integrations from the left navigation and click Connect PagerDuty.
-
Enter a Default email, PagerDuty domain and the API Key from step 2 above. Then click Connect account.
-
Still in the Jeli web app, navigate to Integrations API Tokens. Hit Generate Token on the right and make an API token. Take care to copy it before you leave the page. You could also use an existing Jeli API token if you have one handy.
- Go back to the PagerDuty web app. Navigate to Integrations Workflow Integrations.
- Find the integration name Jeli in the list and click Add Connection. Enter a name for the connection, paste in the API Token you created in the Jeli web app, enter your PagerDuty username, and set your desired access levels.
- After creation, the integration should show a status of "Healthy". Now you can begin using Jeli for your post-incident reviews.
Create a Jeli Post Incident Review
- You can create a Jeli Opportunity from any resolved PagerDuty incident. On a resolved incident, click Jeli Post Incident Review.
- Carefully review the Create a Jeli Post Incident Review prompt and click Accept & Continue to Jeli.
- When prompted, sign in to Jeli using the option Continue with PagerDuty.
- In the Jeli Opportunity, you can review all events from your PagerDuty timeline by clicking Narrative. For more information, please read Opportunities.
FAQ
How can I change the Slack user associated with Postmortems?
To change the associated Slack user account, you'll have to revoke and replace the current Slack authentication only for the Postmortems feature.
Slack User Limitations
It is currently not possible to choose more than one Slack user specifically for the Postmortem feature.
- Within your Slack user account, access the My Authorizations section in the Slack API.
- Locate the PagerDuty Postmortems app under the Application & Scopes column.
- Click the icon to the right of the PagerDuty Postmortems app to remove it only from your chosen Slack user account.
- While editing a postmortem in PagerDuty, click the Connect to Slack button under the Data Sources section to authenticate a different Slack user account to the Postmortems feature.
Is it possible to add images or documents to a postmortem?
It is possible to insert an external link or URL in any postmortem field which hosts the image or document externally. You can insert the link or URL into the section text, timeline entry summary, or in the description of a manual timeline entry within a postmortem. When the link or URL is clicked, then the image or document will open in a new tab. However, it is currently not possible to insert an image directly into a postmortem.
Updated about 2 months ago