Why Incidents Fail to Trigger

Troubleshooting common issues related to incident trigger failure

There are a few reasons why an incident would fail to trigger:

A Service Was Disabled or Placed in Maintenance Mode

When a service is disabled or in maintenance mode, new incidents will not be triggered or created for that service. Because an incident is not created, PagerDuty will not send notifications to the person on call for that service.

Event Rules are Configured to Suppress Certain Alerts

If you have Event Orchestrations, Global Event Rules , or Service Event Rules configured on your account, you may have rules that are set to suppress alerts that meet certain criteria. In those cases, an incident will not be triggered.

For Event Orchestrations, navigate to AIOps Event Orchestration and select the orchestration and the routing rule that contains the action rule that you would like to check. Check to see if there is an action rule that states If event meets [x] condition on [field], suppress alert.

For Global Event Rules, navigate to AIOps Event Rules. Next, check to see if any of your event rules have Suppress selected under Actions performed, or if they have Route to a service selected, but also Suppressing until more than X alerts received within X minutes is selected.

For Service Event Rules, navigate to Services Service Directory, click the name of the service where you expected the incident to trigger, and select the Settings tab.
Under the Automate section you will either see a link to see your event rules or a message saying there are no event rules for this service. Check any event rules to see if Suppress is displayed under Actions Performed.

Email Integration Settings Were Filtering out Emails

If you have regex filters set up on your email integration service, then you will want to check to make sure that your regex filters are not filtering out emails that you want to trigger incidents.

For example, if you have the following regex filter:

Email integration settings

Email integration settings

Then emails with the subject line "PROBLEM" will not trigger incidents in PagerDuty, because they are filtered out based on your regex rules.

Email Management Rules Were Appending Triggers to Existing Incidents

Email integration settings also have email management rule settings. These are distinct from email filters. While email filters determine which emails trigger incidents for your service, incident creation settings determine how emails create new incidents.

If you have either of these two email management rules for your email integration service:

  • Open a new alert only if an open incident does not already exist
  • Open and resolve alerts based on custom rules

...then an incident was most likely appended to an existing incident, which would explain why a new incident was not created. When a trigger is appended to an incident it will appear in the same incident timeline. Learn more about incident creation settings.

No One Was On-call

If no one is on-call on a service's escalation policy, PagerDuty does not have a user to assign an incident to, and we will not create a new incident.

For example, if your escalation policy only has the following schedule associated with it, where one user is on-call from 00:00 - 08:00, if a trigger comes in between 08:01 - 23:59, no new triggers will be created in PagerDuty because no one will be on-call.

To address this, check the escalation policy associated with your service and make sure that someone will be on-call when you need incidents to trigger.

📘

Note

If you try to trigger a new incident through the website while nobody is on-call on the escalation policy associated with that service, you will receive the error Incident could not be assigned.