When a service sends an event to PagerDuty, an alert and corresponding incident is triggered in PagerDuty. Unless the alert has been suppressed, each alert initially has a corresponding incident. Multiple alerts can be aggregated into a single incident for triage. This article discusses how alerts differ from incidents, and how they function to streamline incident handoff between teams, centralize critical information, and reduce notification fatigue.
Note: While all accounts created after November 2016 have alerts and incidents enabled by default, not all accounts have alerts and incidents as a default setting for services. If your account was opened prior to November 2016, you can enable alerts in your account as described here.
Using alerts in PagerDuty
When a third party monitoring tool has an event, an alert is created in PagerDuty. That alert’s incident is what notifies users so that the right person can acknowledge and resolve the incident. By consolidating these alerts into a single incident, you can provide a central incident for your team to respond to and resolve.
Excluding a small handful of exceptions, you can set up alerts for any service that integrates with a third party monitoring tool. Enabling this feature will allow you to merge incidents together to create transparency and clarity when attempting to resolve an incident.
Creating an alert
When a service’s integration has an event, an alert is is created in PagerDuty that is linked to an incident. Notifications are not sent to users based on alerts, which means that you can have one or more alerts under a single incident. That incident will be assigned to users, teams, or schedules on the escalation policy for the service.
Alerts are only created from services using a third party monitoring tool integration, email integration or via our Generic API integration. They cannot be manually created in PagerDuty.
Where an alert is triggered on a service with webhooks enabled, the webhook will send incident actions rather than alert actions. When using a webhook like Slack, you can still ack or resolve your open incidents using the Slack buttons. With other webhook types, merged incidents can no longer be acked via webhook. Instead, the ack action will be noted in the alert timeline.
States of Alert
Alerts can be either triggered, or resolved, but not acknowledged. Alerts can only be created and triggered from third party monitoring tools that are integrated with PagerDuty; they cannot be manually triggered. You can resolve alerts manually or via the API.
If all alerts under an incident are resolved, the incident will likewise become resolved. Conversely, if an incident is resolved then all alerts under the parent incident will be resolved.
Once alerts are enabled for a service, your integration will create an alert and corresponding incident in PagerDuty. All alerts across services may be viewed in the web UI via the Alerts page.
Additionally, any alerts associated with a particular incident may be viewed on the incident's individual page in the web UI. Alerts will be listed in their own section under the Details section.
To expand or collapse the details of the alert such as the message sent from your monitoring tool, click the Show/Hide Details buttons.
account created after November 2016 will have alerts and incidents automatically enabled on services that support alerts and incidents. If you opened your account prior to November 2016, you may not have this feature automatically enabled in your account.
To turn on alerts in your account, open up the settings for a service that you would like to have alerts for. You can find these settings under Configurations → Services. Under the Incident Behaviors section, you can select the option to Create alerts and incidents.
On accounts without alerts and incidents automatically enabled, you will need to enable alerts and incidents on each service that you would like this functionality to extend to. Enabling this feature of one service will not enable it on all services.
When enabled the
incident_key is deprecated and becomes
null. De-duplication will be based on on the
alert_key instead of the
incident_key. This may cause issues with some bidirectional integrations. Additionally, the summary of the number of alerts is found using the
alert_count. You can read more on this in our API Documentation.
Once alerts are enabled and for a service, you can review our knowledge base article on merging alerts into a single incident.
Alerts and bidirectional integrations
With PagerDuty’s new alerts and incidents functionality, most services can be migrated to create alerts and incidents. However, bi-directional integrations that allow PagerDuty to send information back out to a tool cannot support alerts and incidents.
Bi-directional integrations help keep outside monitoring tools in sync with PagerDuty acknowledge or resolve actions by sending a webhook when an incident’s state is updated. These third-party tools depend on
incident_key being present in the webhook payload to update events on their end. With the release of alerts and incidents, webhook payloads will no longer contain
incident_key when the incident contains an alert; instead the field will show as
Bidirectional integrations with this limitation include the following:
CA UIM – Email
Live Call Routing
- Nagios and forks of Nagios: Check_MK, Icinga, Nagios XI, Groundwork Nagios, Opsview
Slack to PagerDuty
If you are using any of the above bidirectional integration on a service, you will not be able to activate alerts and incidents. The option to do so on the Edit Service page will be grayed out. Additionally, if two services are present on one integration and one of the services is not compatible with alerts and incidents, the option to enable that will also be grayed out.