Dynamic Notifications
Add severity levels to incidents to support effective incident triage
The Dynamic Notifications feature allows users to generate alerts with severity fields. When an incident is generated from an alert, its severity field can be used to control incident urgency and how responders are notified. This feature reduces noise and ensures responders can focus on the incidents that matter most.
Pricing Plans
All pricing plans, with the exception of some legacy and specially negotiated plans, have access to Dynamic Notifications. Please contact our Sales team if you'd like to switch to a plan with Dynamic Notifications.
Event/Alert Severity Levels
Alerts in PagerDuty can be generated with a severity field. These severity values can be directly provided from the triggering monitoring tool, or set using Event Orchestration.
When an incident is generated from an alert, the alert’s severity field is used to determine the urgency level. The values of this field must be one of the following: critical
, error
, warning
, or info
.
Case Sensitivity
Severity fields are case sensitive. For example, the severity field
Info
(instead ofinfo
) will produce an error:'payload.severity' is invalid (must be one of the following: 'critical', 'warning', 'error' or 'info')
.
Severity and Urgency Mapping
Alert Severity | Description | Default Incident Urgency | Incident Behavior |
---|---|---|---|
critical | A failure in the system's primary application. | High | Uses high-urgency notification rules and escalates if not acknowledged. |
error | Any error which is fatal to the operation, but not the service or application. | High | Uses high-urgency notification rules and escalates if not acknowledged. |
warning | May indicate that an error will occur if action is not taken. | Low | Uses low-urgency notification rules and cannot be escalated. |
info | Normal operational messages that require no action. | Low (if appended to an incident; we recommend suppressing info alerts) | Uses low-urgency notification rules and cannot be escalated. |
Unknown | Automatically chosen when a monitoring tool is not setting, or cannot set the severity. | High | Uses high-urgency notification rules and escalates if not acknowledged. |
Severity to urgency mappings are hard-coded (as shown below). Any incoming events that do not contain a Severity will default to high-urgency.
Configure Dynamic Notifications
Required User Permissions
Users with the following roles can configure Dynamic Notifications:
- User
- Admin
- Manager base roles and team roles
- Manager team roles can only manage services associated with their team.
- Global Admin base roles
- Account Owner
- Navigate to Services Service Directory and select your preferred service.
- Select the Settings tab and click Edit to the right of the Assign and Notify section.
- In the How should responders be notified? dropdown, select Dynamic notifications based on alert severity.
- Click Save Changes.
Any alert-created incidents on the service will now determine their notification urgency according to the severity of their alerts. When an alert is added to an incident, or when an alert’s severity changes, it will also update its corresponding incident’s notification urgency, but only in an upward direction. For example, an alert changing from Warning to Critical will change the associated incident from Low to High, but another change from Critical to Warning will not reduce the urgency of the incident.
Non-alert incidents, such as those manually triggered or triggered through the Create Incidents API, will always have their urgency set by the service’s setting.
Defined Support Hours
Dynamic Notifications can also be used in relation to defined support hours on a service. Users can set specific support hours, and decide how they want to be notified inside and outside of this time window. When support hours end, users also have the option to either leave incidents at their current urgency, or raise the urgency of all triggered incidents for the service to High.
To configure defined support hours:
- Navigate to Services Service Directory and select your preferred service.
- Select the Settings tab and click Edit to the right of the Assign and Notify section.
- In the How should responders be notified? dropdown, select Based on support hours.
- Select the days of the week, hours and time zone for your support hours. Next, under During support hours, use, select what type of notification urgency you would like during support hours. Under Outside support hours, use, select what type of notification urgency you would like outside of support hours. You may also optionally choose to check the box next to Raise urgency of unacknowledged incidents to high when support hours start.
- Click Save Changes.
Upgrade Incident Notification Urgency
If multiple trigger events come in for the same alert, the alert will upgrade its severity value to the most severe value. For example, if an alert was initially triggered by a Warning event, but then a Critical event comes in, the alert will now be Critical, and the associated incident will change from Low urgency to High urgency. If a subsequent Info event comes in, the incident will remain High urgency. It will not downgrade to Low.
There are three ways in which multiple alerts can be bundled into a single incident:
- Using our Alert Grouping feature. Here, users can group alerts for a set period of time, or until the incident resolves. While grouping is activated, subsequent alerts will roll up under the single incident. The incident will adopt the notification urgency that corresponds with the most severe alert that is grouped with it.
- By manually merging the alerts’ parent incidents. Below is an example of two alerts that have been merged into a single incident. In this example, the urgency of each alert was raised to High by the merge:
- Using an alert
dedup_key
. With this method, a subsequent alert trigger will replace an existing alert.
PagerDuty Common Event Format
For integrations already mapped to the PagerDuty Common Event Format (PD-CEF), alerts of varying severities may be generated by default. For most integrations, however, alerts are generated as critical by default, and Event Orchestration must be used to set the desired severity.
Recommended Practices
- For immediately actionable events, set a
critical
orerror
severity: This will ensure that any associated incidents use high-urgency notification channels and escalate if not acknowledged. - For actionable events that do not require immediate attention, set a
warning
severity: This will create a low-urgency incident that does not escalate. - For non-actionable events, consider suppressing them in addition to setting an
info
severity: Non-actionable events do not require incident response, so in most cases you will want to suppress these.
FAQ
How does the Dynamic Notifications feature interact with Incident Priority?
Incident Priority and Incident Urgency are separate properties that today, do not influence each other. Currently, Priority can be set manually by users on an existing incident, during incident creation via manual creation, with the Incident Create API or it can be set using Event Orchestration.
What if an alert does not have a severity attached?
Alerts created prior to November 2016 may not have severity information attached. However, since then, all alerts are generated with severity, and default to Critical if the severity is not specified by the emitting system, by event transformer, or by Event Orchestration. Events sent to the Events API v2 must have severity specified.
What about email integrations?
Service level email integrations will also not send in severity. This functionality may be available in the future.
Tip
You can send events to Event Orchestration's email integration key to add severity and incident priority.
Can I customize the mapping of severity to urgency?
Currently mappings are not customizable. We intend to build this capability into the platform in future iterations of this feature.
How do I set urgency if monitoring tools do not send it to PagerDuty?
You can use Event Orchestration to set severities for incoming alerts on a service.
How does the Dynamic Notifications feature relate to PagerDuty CEF?
Integrations mapped to CEF will automatically send in severity. Users can always send in severity information using the Events API v2.
How does the Dynamic Notifications feature relate to Alert Grouping?
Users can group alerts into a single incident, either over a specific time period, or for as long as the incident is open. As new alerts bundle into an incident, we will upgrade the incident urgency to the most severe alert. We will never downgrade an incident's urgency.
Will Incident Log Entries Update?
Yes, we update the ILEs to indicate when incident urgency is set by severity, and when incident urgency is upgraded due to alert severity.
Updated 3 months ago