There are a few reasons why an incident would fail to trigger:
- A Service Was Disabled or Placed in Maintenance Mode
- Event Rules Were Configured to Suppress Certain Alerts
- Email Integration Settings Were Filtering out Emails
- Email Management Rules Were Appending Triggers to Existing Incidents
- No One Was On-call
When a service is disabled or in maintenance mode, new incidents will not be triggered or created for that service. Because an incident is not created, PagerDuty will not send notifications to the person on call for that service.
If you have Service Event Rules, Global Event Rules, or Event Orchestrations configured on your account, you may have rules that are set to suppress alerts that meet certain criteria, and in those cases, an incident will not be triggered.
For Service Event Rules, navigate to Services Service Directory, click the name of the service where you expected the incident to trigger, and select the Settings tab.
Under the Automate section you will either see a link to see your event rules or a message saying there are no event rules for this service. Check any event rules to see if Suppress is displayed under Actions Performed.
For Global Event Rules, navigate to Automation Event Rules. Next, check to see if any of your event rules have Suppress selected under Actions performed, or if they have Route to a service selected, but also Suppressing until more than X alerts received within X minutes is selected.
For Event Orchestrations, navigate to Automation Event Orchestration and select the orchestration and the routing rule that contains the action rule that you would like to check. Check to see if there is an action rule that states “If event meets [x] condition on [field], suppress alert”.
If you have regex filters set up on your email integration service, then you will want to check to make sure that your regex filters are not filtering out emails that you want to trigger incidents.
For example, if you have the following regex filter:
Then emails with the subject line "PROBLEM" will not trigger incidents in PagerDuty, because they are filtered out based on your regex rules.
- View some regular expression examples
- Test your regular expressions with Regex101 (Use the Golang flavor with the
sflags to see exactly how PagerDuty will interpret your regex).
Email integration settings also have email management rule settings. These are distinct from email filters. While email filters determine which emails trigger incidents for your service, incident creation settings determine how emails create new incidents.
If you have either of these two email management rules for your email integration service:
- Open a new alert only if an open incident does not already exist
- Open and resolve alerts based on custom rules
...then an incident was most likely appended to an existing incident, which would explain why a new incident was not created. When a trigger is appended to an incident it will appear in the same incident timeline. Learn more about incident creation settings.
If no one is on-call on a service's escalation policy, PagerDuty does not have a user to assign an incident to, and we will not create a new incident.
For example, if your escalation policy only has the following schedule associated with it, where one user is on-call from 00:00 - 08:00, if a trigger comes in between 08:01 - 23:59, no new triggers will be created in PagerDuty because no one will be on-call.
To address this, check the escalation policy associated with your service and make sure that someone will be on-call when you need incidents to trigger.
If you try to trigger a new incident through the website while nobody is on-call on the escalation policy associated with that service, you will receive the error Incident could not be assigned.
Updated about 2 months ago