Notification fatigue can be assuaged with the following actions:
incident_key in your API call
If your service is set up as a Generic API service type and multiple incidents are triggering for the same issue your team will be notified for each duplicate incident. To group these incidents you will want to include
incident_key in your parameters for triggering incidents.
PagerDuty de-duplicates incidents based on the
incident_key parameter; this identifies the incident to which a trigger event should be applied. If there are no open (unresolved) incidents with this key, a new incident will be created.
If there is already an open incident with a matching key, this event will be appended to that incident's alert log as an additional Trigger log entry.
If the event key field isn't provided, PagerDuty will automatically open a new incident with a unique key.
Change your email management settings
If you have an email integration you can change your incident creation settings so that incidents are only triggered under certain conditions.
There are four types of email management settings:
Open a new alert/incident for each trigger email: Each email sent to the service's email address will open a new incident.
Open a new alert/incident for each new trigger email subject:Incidents are de-duped based on the subject of the trigger emails. For example, if two emails with the same subject are sent to this service's email address, the first opens a new incident, and the second is appended to this incident.
Open a new alert/incident only if an open incident does not already exist: An email sent to the service's email address will only open a new incident if they service has no open incidents; otherwise, the email will be appended to the open incident.
Create and resolve alerts/incidents based on custom rules: Use regular expressions to parse incident triggers and resolves.
The last three incident creation settings are ones that allow alert/incident de-duplication and reduce the number of notifications being sent.
To change the incident creation settings for your email integration service:
Go to the Configuration menu and select Services
Click on the name of the service that houses the integration, then select the Integrations tab. You can edit the integration by selecting Edit from the settings cog, or from the integration details page accessible by clicking the name of the integration.
Select the appropriate email management setting from the four options provided.
Click Save changes.