Incidents: Creating transparency across teams

Follow

Sometimes, it's not just the people on-call who need to know about an incident. Managers, Support Teams, Sales Teams, and other teams may also want or need to be aware of incidents that aren't necessarily assigned to them.

Whether these teams are in PagerDuty or not, there are many ways in which you can keep others informed about incidents and events that are triggered in PagerDuty

Here are some of our recommendations and best practice tips for creating transparency in your organization:

Use webhooks to email multiple teams

Webhooks are tied to services and they can send out HTTP callbacks when different events happen to an incident triggered on a particular service.

Many teams set up a webhook on their service so that an email is sent any time an incident is triggered.

This is ideal if you want to email a distribution list or list of users who are not users in PagerDuty and do not have access or receive alerts from PagerDuty.

Zapier or a similar service can be used to consume PagerDuty webhooks and send an email based on these webhooks.

Push incidents into your collaboration/chat tool

Integrate PagerDuty with Slack, Hipchat, Flowdock, Socialcast or any of our other Collaboration/Chat integration partners. This is a great way to let teams know when incidents are triggered, acknowledged, and resolved in real time, without having to manually alert everybody

 

Publish incidents on an internal status page

Use a status page app such as StatusPage.io or Status.io. These do a great job of grouping PD services into components so stakeholders can understand, in language that's relevant to them, what is up and working against what is down (and subscribe to email alerts about these components).

 

Create internal dashboards with our API

You can use our REST API to create internal dashboards to display open incidents.

 

Add distribution lists to your escalation policies

An alternative to using webhooks to email distribution lists (as described above), you can create a user with email distribution lists as their email contact method(s) and notification rule(s), and add that user to the appropriate level of your escalation policies.

When an incident is triggered against a service that is using that escalation policy, notifications would go out to all of the email addresses listed as a notification rule under that user profile.

Note: You may want to create a separate distribution list user per team you wish to notify, as not everybody necessarily wants notifications about incidents outside of their team.

 

Add managers to escalation policies

To increase visibility into alerts on certain services and escalation policies, a manager (or any other user/schedule) can be added to the first-level of an escalation policy along with the first user/schedule on-call so that the manager is notified about incidents that are triggered.

Additionally, if the manager added status update notification rules to their profile, they can gain even more visibility into incidents with alerts, but setting up notifications when incidents have been acknowledged, resolved, or escalated.

If this manager is not the primary responder for incidents though, it is recommended that they do not acknowledge, resolve, or escalate incidents before the on-call user has a chance to see it.

Have more questions? Submit a request

Comments