There are a few reasons why an incident would not be triggered or created in PagerDuty:
When a service is disabled or in maintenance mode, new incidents will not be triggered or created for that service. Because an incident is not created, PagerDuty will not send notifications to the person on-call for that service.
If you have Service Event Rules or Global Event Rules configured on your account, you may have event rules that are set to suppress alerts that meet certain criteria, and in those cases, an incident will not be triggered.
For Service Event Rules, navigate to Configuration Services, click the name of the service where you expected the incident to trigger, and select the Event Rules tab. Check any event rules to see if Suppress is selected under Actions Performed:
For Global Event Rules, navigate to Configuration Event Rules. Next, check to see if any of your event rules have Do not route alert to a service selected under Alert Behavior, or if they have Route to a service selected, but under Incident Behavior, Suppress and do not create an incident is selected:
If you have regex filters set up on your email integration service, then you will want to check to make sure that your regex filters are not filtering out emails that you want to trigger incidents.
For example, if you have the following regex filter:
Then emails with the subject line "PROBLEM" will not trigger incidents in PagerDuty, because they are filtered out based on your regex rules.
Email integration settings also have email management rule settings. These are distinct from email filters. While email filters determine which emails trigger incidents for your service, incident creation settings determine how emails create new incidents.
If you have either of these two email management rules for your email integration service:
- Open a new alert only if an open incident does not already exist
- Open and resolve alerts based on custom rules
...then an incident was most likely appended to an existing incident, which would explain why a new incident was not created. Learn more about incident creation settings.
When a trigger is appended to an incident it will look like the following. The appended trigger is highlighted in green. Notice that both triggers are appended to the same incident, appearing in the same incident timeline.
If no one is on-call on a service's escalation policy, PagerDuty does not have a user to assign an incident to, and we will not create a new incident.
For example, if your escalation policy only has the following schedule associated with it, where one user is on-call from 00:00 - 08:00, if a trigger comes in between 08:01 - 23:59, no new triggers will be created in PagerDuty because no one will be on-call.
To address this, check the escalation policy associated with your service and make sure that someone will be on-call when you need incidents to trigger.
If you try to trigger a new incident through the website while nobody is on-call on the escalation policy associated with that service, you will receive the error Incident could not be created.