When a service sends an event to PagerDuty, an alert and corresponding incident is triggered in PagerDuty. Unless the alert has been suppressed, each alert initially has a corresponding incident. Multiple alerts can be aggregated into a single incident for triage, functioning to streamline incident handoff between teams, centralize critical information, and reduce notification fatigue.
When a third party monitoring tool has an event, an alert is created in PagerDuty. That alert’s incident is what notifies users so that the right person can acknowledge and resolve the incident. By consolidating these alerts into a single incident, you can provide a central incident for your team to respond to and resolve.
Excluding a small handful of exceptions, you can set up alerts for any service that integrates with a third party monitoring tool. Enabling this feature will allow you to merge incidents together to create transparency and clarity when attempting to resolve an incident.
When a service’s integration has an event, an alert is created in PagerDuty that is linked to an incident. Notifications are not sent to users based on alerts, which means that you can have one or more alerts under a single incident. That incident will be assigned to users, teams, or schedules on the escalation policy for the service.
Alerts are only created from services using a third party monitoring tool integration, email integration or via our Generic API integration. They cannot be manually created in PagerDuty.
Where an alert is triggered on a service with webhooks enabled, the webhook will send incident actions rather than alert actions. When using a webhook like Slack, you can still ack or resolve your open incidents using the Slack buttons. With other webhook types, merged incidents can no longer be acked via webhook. Instead, the ack action will be noted in the alert timeline.
Alerts can be either triggered, or resolved, but not acknowledged. Alerts can only be created and triggered from third party monitoring tools that are integrated with PagerDuty; they cannot be manually triggered. You can resolve alerts manually or via the API.
If all alerts under an incident are resolved, the incident will likewise become resolved. Conversely, if an incident is resolved then all alerts under the parent incident will be resolved.
Once alerts are enabled for a service, your integration will create an alert and corresponding incident in PagerDuty. All alerts across services may be viewed in the web UI via the Alerts page on the main menu.
Additionally, any alerts associated with a particular incident may be viewed on the incident's individual page in the web UI. Alerts will be listed in their own section under the Details section.
To expand or collapse the details of the alert such as the message sent from your monitoring tool, click the Show/Hide Details buttons.
Alerts are only created for inbound events submitted via one of our events APIs (Events API (v1) or Events API (v2)), or through any of the majority of integrations that utilize an events API based integration. Incidents created via any of the following processes would have no alerts associated with them, and thus, alert rules, suppression and other features of alerts will not be applicable:
- Email integrations
- Incident creation via web UI ("New Incident" button)
- Incident Creation via REST API
You might not need to manually enable alerts
While all accounts created after November 2016 have alerts and incidents enabled by default, not all accounts have alerts and incidents as a default setting for services. If your account was opened prior to November 2016, you can enable alerts in your account as described below.
To turn on alerts in your account, open up the settings for a service that you would like to have alerts for. You can find these settings under Configurations → Services. Under the Incident Behaviors section, you can select the option to Create alerts and incidents.
On accounts without alerts and incidents automatically enabled, you will need to enable alerts and incidents on each service that you would like this functionality to extend to. Enabling this feature of one service will not enable it on all services.
When enabled, the
incident_key is deprecated and becomes
null. De-duplication will be based on on the
alert_key instead of the
incident_key. This may cause issues with some bidirectional integrations. Additionally, the summary of the number of alerts is found using the
alert_count. You can read more on this in our API Documentation.
Once alerts are enabled and for a service, you can review our knowledge base article on merging alerts into a single incident.
Alerts Table Accesibility
The alerts table is accessible via the Alerts tab in the navigation bar. This tab is only visible on accounts with alerts and incidents enabled.
The alerts table, available in our web UI, allows you to view alerts according to your preferences, helping you locate what you're looking for quickly and efficiently.
Alerts are filtered via the columns that appear in the alerts table. The type of filter (radio button or search) varies based on column data type.
The columns Severity, Summary, Source, Class, Component, and Group all map to PD-CEF fields, which can be easily included in your events with our new Events API v2. PD-CEF is currently available for Splunk, AWS CloudWatch, DataDog, Nagios, Sensu and Zabbix integrations.
The Status column indicates whether your alerts are triggered and/or resolved, suppressed and/or actionable.
Related Incident points to the incident your alerts are grouped under, and Service and Integration point to the service and integration your alerts are associated with.
Using the Customize Columns button, users may show or hide the Related Incident, Service, Integration, Source, Class, Component and Group columns. Status, Severity, Summary and Created columns are automatically enabled, and cannot be removed from your view, which is why they appear in gray under the Customize Columns button.
Related Incident, Service, Integration and Source are automatically enabled, but can be hidden.
Search filtering is enabled on the Summary, Related Incident, Integration, Source, Class, Component, and Group columns. Partial matches will be displayed, for example searching Prod will display prod04 and Prod03 in the results.
A majority of the columns in the table have sort capability. The default sort setting is by Status (Triggered, Triggered Suppressed, Resolved, Resolved Suppressed). Sortable columns will have a pair of arrows to the right of the column name. Click on the arrows to sort your alerts based on this column. Active sort filters will appear as a darker grey than the columns surrounding them.
With the active filters bar you will never lose track of which filters are currently on. Column tags populate the bar, and filter icon highlight blue when filters are active. This bar appears at the top of your table.
Table filters can be cleared from within the Active Filters Bar by clicking the x next to each individual filter, or by clicking Clear all table filters. In the event that no results are found when a search filter is input, this may be removed by clicking the search icon a second time and clicking Clear filter.
With PagerDuty’s new alerts and incidents functionality, most services can be migrated to create alerts and incidents. However, bi-directional integrations that allow PagerDuty to send information back out to a tool cannot support alerts and incidents.
Bi-directional integrations help keep outside monitoring tools in sync with PagerDuty acknowledge or resolve actions by sending a webhook when an incident’s state is updated. These third-party tools depend on
incident_key being present in the webhook payload to update events on their end. With the release of alerts and incidents, webhook payloads will no longer contain
incident_key when the incident contains an alert; instead the field will show as null.
Bidirectional integrations with this limitation include the following:
- CA UIM
- CA UIM – Email
- JIRA Software
- Nagios and forks of Nagios: Check_MK, Icinga, Nagios XI, Groundwork Nagios, Opsview
- ServiceNow v3.2.1 (v3.5 and beyond are unaffected)
- Slack to PagerDuty
If you are using any of the above bidirectional integration on a service, you will not be able to activate alerts and incidents. The option to do so on the Edit Service page will be grayed out. Additionally, if two services are present on one integration and one of the services is not compatible with alerts and incidents, the option to enable that will also be grayed out.
Updated about a month ago