Methods for alert noise reduction
Event management reduces notification fatigue and helps your team focus on tackling the right issues at the right time. The following methods are effective ways to prevent notification fatigue:
- Deduplicate incidents on services with API integrations.
- Adjust your email management settings on services with email integrations.
- Use Dynamic Notifications to set alert severity.
- Use Rulesets or Event Orchestrations to automatically deduplicate or suppress alerts.
For services with API integrations, if multiple incidents are triggering for the same issue, your team will be notified for each duplicate incident. To group these incidents, you will want to include
dedup_key (Events API v2) or
incident_key (Events API v1) in your parameters for triggering incidents.
Required Service Settings
In order to deduplicate incidents, you must have your service set to create both alerts and incidents.
To do this, follow our instructions to enable alerts on a service and selecting Create both alerts and incidents.
incident_key identifies incidents where a trigger event should be applied, and deduplicates subsequent incidents:
- If there are no open incidents with this key, a new incident will be created.
- If there is an open incident with a matching key, the new event will be appended to that incident's Alerts log as an additional Trigger log entry.
- If two events with the same
incident_keyare sent to two different integrations on the same service, they will not be deduplicated and will trigger different incidents instead.
If the event key field isn't provided, PagerDuty will automatically open a new incident with a unique key.
To learn more about how to deduplicate incidents, please visit our Developer Documentation.
Adjust your Email Management Settings
If you have an email integration you can change your incident creation settings so that incidents are only triggered under certain conditions.
The following incident creation settings allow for alert/incident de-duplication and reduce the number of notifications being sent:
- Open a new alert/incident for each new trigger email subject: Incidents are deduplicated based on the subject of the trigger emails. For example, if two emails with the same subject are sent to this service's email address, the first opens a new incident, and the second is appended to the incident.
- Open a new alert/incident only if an open incident does not already exist: An email sent to the service's email address will only open a new incident if the service has no open incidents; otherwise, the email will be appended to the open incident.
- Create and resolve alerts/incidents based on custom rules: Use regular expressions to parse incident triggers and resolves.
To configure these settings, please visit our article on email filters and email management rules.
Dynamic notifications allow you to generate alerts with a severity field. When an incident is generated from an alert, the alert’s severity field allows you to filter how responders are notified, if they are notified at all. Please read our Dynamic Notifications article for configuration instructions and more information.
Event Orchestration and Rulesets
Event Orchestration and Rulesets are features that allow users to define automated actions to take on alerts created by services, based on conditions that apply to information in the inbound events' payloads.
- Event Orchestration: Event Orchestrations allow you to build nested rules based on the events coming from the integrated tool. Event Orchestration is only available with the PagerDuty AIOps add-on, or with Digital Operations plans.
- Rulesets: Rulesets allow you to build different routing rules based on the events coming from the integrated tool.
Both features can perform actions such as deduplicating or automatically suppressing alerts altogether.
Suppression, as opposed to setting alert severity, allows you to send events to PagerDuty without triggering any notifications. Suppressed alerts are stored in PagerDuty and are available for forensics, analysis, and context, but do not create incidents. Suppressed alerts can be viewed in the Alerts table.
To suppress an alert if it matches a given set of conditions, select Suppress as the action to take when you are building your rulesets or orchestrations.
View Suppressed Alerts
Suppressed alerts can be viewed in the Alerts Table under Incidents Alerts.
Suppressed alerts are filtered out of the incidents dashboard by default, including the incidents page for the service where they were triggered. Moreover, because suppressed events do not trigger incidents, they will not be visible in the mobile app.
Suppressed Event Example
Below is an example of a suppressed alert. It looks very similar to other alerts, but has Triggered (Suppressed) in the Current Status field and is not assigned/assignable.
What is the difference between deduplication and suppression?
- Deduplication: Alerts with the same incident key are grouped together into the same incident and do not generate multiple notifications.
- Suppression: Alerts are filtered by the conditions set in your rulesets, and those that match your criteria are suppressed and stored in PagerDuty to be available for forensics, analysis, and context. The major difference between suppressed and deduplicated alerts is that suppressed alerts do not create incidents. Suppressed alerts can be viewed in the Alerts table. It should also be noted that suppressed alerts can be deduplicated.
Will suppressed alerts still show up on a service with no one on call?
If you send an event to a service where no one is on call, the event will not create an incident. However, if the event is meant to be suppressed, it will still go to the service if no one is on call.
Updated 26 days ago