Event Management
Methods for alert noise reduction
Event management reduces notification fatigue and helps your team focus on tackling the right issues at the right time. The following methods are effective ways to prevent notification fatigue:
- Deduplicate incidents on services with API integrations.
- Adjust your email management settings on services with email integrations.
- Use Dynamic Notifications to set alert severity.
- Use Event Orchestration to automatically deduplicate or suppress alerts.
Deduplicate Incidents
For services with API integrations, if multiple incidents are triggering for the same issue, your team will be notified for each duplicate incident. To group these incidents, you will want to include dedup_key
(i.e., Events API v2) or incident_key
(i.e., Events API v1) in your parameters for triggering incidents.
The dedup_key
/incident_key
is a case-sensitive identifier that indicates where a trigger event should be applied, and deduplicates subsequent alerts/incidents:
- If there are no open incidents with this key, a new incident will be created.
- If there is an open incident with a matching key, the new event will be appended to that incident's Alerts log as an additional Trigger log entry.
- If two events with the same
dedup_key
/incident_key
are sent to two different integrations on the same service, they will not be deduplicated and will trigger different incidents instead.
If the event key field isn't provided, PagerDuty will automatically open a new incident with a unique key.
To learn more about how to deduplicate incidents, please visit our Developer Documentation.
Adjust your Email Management Settings
If you have an email integration you can change your incident creation settings so that incidents are only triggered under certain conditions.
The following incident creation settings allow for alert/incident de-duplication and reduce the number of notifications being sent:
- Open a new alert/incident for each new trigger email subject: Incidents are deduplicated based on the subject of the trigger emails. For example, if two emails with the same subject are sent to this service's email address, the first opens a new incident, and the second is appended to the incident.
- Open a new alert/incident only if an open incident does not already exist: An email sent to the service's email address will only open a new incident if the service has no open incidents; otherwise, the email will be appended to the open incident.
- Create and resolve alerts/incidents based on custom rules: Use regular expressions to parse incident triggers and resolves.
To configure these settings, please visit our article on email filters and email management rules.
Dynamic Notifications
Dynamic notifications allow you to generate alerts with a severity field. When an incident is generated from an alert, the alert’s severity field allows you to filter how responders are notified, if they are notified at all. Please read Dynamic Notifications for configuration instructions and more information.
Event Orchestration
Event Orchestration is a feature that allows users to define automated actions, based on conditions that apply to the information in an inbound event's payload. Event Orchestration can perform actions such as deduplicating or automatically suppressing alerts altogether. Please read Event Orchestration for more information.
Alert Suppression
Availability
Alert Suppression is available on accounts with AIOps, or Enterprise for Incident Management and Digital Operations (Legacy) pricing plans. Please contact our Sales team to upgrade to a plan with this feature.
Alert suppression, as opposed to setting an alert's severity, allows you to send events to PagerDuty without triggering any notifications. Suppressed alerts are stored in PagerDuty and are available for forensics, analysis, and context, but do not create incidents. You can view suppressed alerts in the Alerts table.
To suppress an alert when it matches a given set of conditions, select Suppress as the action to take when you are configuring Event Orchestrations.
View Suppressed Alerts
Suppressed alerts can be viewed in the Alerts Table under Incidents Alerts.
Suppressed alerts are filtered out of the incidents dashboard by default, including the incidents page for the service where they were triggered. Moreover, because suppressed events do not trigger incidents, they will not be visible in the mobile app.
Suppressed Event Example
Below is an example of a suppressed alert. It looks very similar to other alerts, but has Triggered (Suppressed) in the Current Status field and is not assigned/assignable.
FAQ
What is the difference between deduplication and suppression?
- Deduplication: Alerts with the same incident key are grouped together into the same incident and do not generate multiple notifications.
- Suppression: Alerts are filtered by the conditions set in your Event Orchestrations, and those that match your criteria are suppressed and stored in PagerDuty to be available for forensics, analysis, and context. The major difference between suppressed and deduplicated alerts is that suppressed alerts do not create incidents. Suppressed alerts can be viewed in the Alerts table. It should also be noted that suppressed alerts can be deduplicated.
Will suppressed alerts still show up on a service with no one on call?
If you send an event to a service where no one is on call, the event will not create an incident. However, if the event is meant to be suppressed, it will still go to the service if no one is on call.
Updated about 1 month ago