Why Incidents Fail to Trigger
Troubleshooting common issues related to incident trigger failure
There are a few reasons why an incident would fail to trigger:
- A Service Was Disabled or Placed in Maintenance Mode
- Event Rules Were Configured to Suppress Certain Alerts
- Email Integration Settings Were Filtering out Emails
- Email Management Rules Were Appending Triggers to Existing Incidents
- No One Was On-call
A Service Was Disabled or Placed in Maintenance Mode
When a service is disabled or in maintenance mode, new incidents will not be triggered or created for that service. Because an incident is not created, PagerDuty will not send notifications to the person on call for that service.
Event Rules are Configured to Suppress Certain Alerts
If you have Event Orchestrations, Global Event Rules , or Service Event Rules configured on your account, you may have rules that are set to suppress alerts that meet certain criteria. In those cases, an incident will not be triggered.
For Event Orchestrations, navigate to AIOps Event Orchestration and select the orchestration and the routing rule that contains the action rule that you would like to check. Check to see if there is an action rule that states If event meets [x] condition on [field], suppress alert.
For Global Event Rules, navigate to AIOps Event Rules. Next, check to see if any of your event rules have Suppress selected under Actions performed, or if they have Route to a service selected, but also Suppressing until more than X alerts received within X minutes is selected.
For Service Event Rules, navigate to Services Service Directory, click the name of the service where you expected the incident to trigger, and select the Settings tab.
Under the Automate section you will either see a link to see your event rules or a message saying there are no event rules for this service. Check any event rules to see if Suppress is displayed under Actions Performed.
Email Integration Settings Were Filtering out Emails
If you have regex filters set up on your email integration service, then you will want to check to make sure that your regex filters are not filtering out emails that you want to trigger incidents.
For example, if you have the following regex filter:
Then emails with the subject line "PROBLEM" will not trigger incidents in PagerDuty, because they are filtered out based on your regex rules.
- View some regular expression examples
- Test your regular expressions with Regex101 (Use the Golang flavor with the
m
ands
flags to see exactly how PagerDuty will interpret your regex).
Email Management Rules Were Appending Triggers to Existing Incidents
Email integration settings also have email management rule settings. These are distinct from email filters. While email filters determine which emails trigger incidents for your service, incident creation settings determine how emails create new incidents.
If you have either of these two email management rules for your email integration service:
- Open a new alert only if an open incident does not already exist
- Open and resolve alerts based on custom rules
...then an incident was most likely appended to an existing incident, which would explain why a new incident was not created. When a trigger is appended to an incident it will appear in the same incident timeline. Learn more about incident creation settings.
No One Was On-call
If no one is on-call on a service's escalation policy, PagerDuty does not have a user to assign an incident to, and we will not create a new incident.
For example, if your escalation policy only has the following schedule associated with it, where one user is on-call from 00:00 - 08:00, if a trigger comes in between 08:01 - 23:59, no new triggers will be created in PagerDuty because no one will be on-call.
To address this, check the escalation policy associated with your service and make sure that someone will be on-call when you need incidents to trigger.
Note
If you try to trigger a new incident through the website while nobody is on-call on the escalation policy associated with that service, you will receive the error Incident could not be assigned.
Updated 2 months ago