Event Rules define automated actions to take on alerts created by services, based on conditions that apply to information in the inbound events' payloads. Event Rules can perform such actions as setting the severity of the resulting alert, or automatically suppressing the alert altogether.
Event Rules can be used on services that use vendor-specific and API-based integrations. For email integrations, we recommend utilizing email management rules.
- Severity versus Urgency
- Configure Event Rules For a Service
- Event Rule Behavior
- Setting Conditions for Event Rules
- Suppressing Alerts
Severity versus Urgency
In addition to suppressing alerts, event rules can adjust severity on alerts to be different than what they would otherwise be, from the original events sent by monitoring tools.
Severity, a property of alerts, has to do with the level of impact to a specific application or unit of infrastructure. Urgency, a property of incidents, pertains to the scope and/or scale of the human response.
We recommend applying a "critical" event rule to anything that you would like to be treated as being of high urgency - these are what you'd like your team to address most aggressively. A "warning" Event Rule can be applied to one that would be considered low priority, and so would still trigger an alert, but doesn't merit a high urgency setting. An Event Rule with "info," can be used for an event that you would like to be suppressed.
Configure Event Rules For a Service
To use event rules on a service, that service must first be set to create alerts and incidents for inbound events. To do this, go to Configuration → Services, edit the service and locate the Incident Behavior section at the bottom of the Edit Service page:
Event Rules then can be added from the individual service page; the Event Rules tab is located on the far right:
Event Rule Behavior
Rules are evaluated in the order they appear in the Event Rules configuration tab, top to bottom. The first matching rule is applied, and then execution stops. If no rule matches, the event is processed ordinarily, as if there were no rules configured.
The order of rules can be controlled by dragging and dropping the rules in the Event Rules page.
If an event rule is deleted while a suppressed alert is in a "triggered" state, further alerts with the same alert key will continue to deduplicate until the alert is resolved.
Setting Conditions for Event Rules
When creating a rule, first set the conditions under which the rule will apply. For example:
In the above example, the rule will apply to an inbound event if the value of its «Source» field is exactly prod05-nginx.
Event rule conditions can be configured for any of the following PagerDuty Common Event Format (PD-CEF) fields:
- Custom Details
Once you have selected a field to examine, select a condition, such as equals or contains, and (if applicable) enter a value to compare the field against. In addition to basic comparison, regular expressions can be used. Note, the RE2 flavor of RegEx is used in evaluation of expressions.
Suppression, as opposed to setting the severity of alerts, allows you to send events to PagerDuty without triggering any notifications. Suppressed alerts are stored in PagerDuty and available for forensics, analysis, and context, but do not create incidents. Suppressed alerts can be viewed in the alerts list as well as the Infrastructure Health Application.
To suppress an alert if it matches a given set of conditions, select «Suppress» as the action to take:
Viewing Suppressed Alerts
Suppressed alerts are filtered out of the incidents dashboard by default, including the incidents page for the service on which the suppressed alerts were triggered. Moreover, because suppressed events do not trigger incidents, they will not be visible in the mobile app.
They can be viewed in the Alerts tab at the top of your screen, between Incidents and Configuration:
Here is an example of a suppressed event. It looks very similar to other alerts, but has "Suppressed" in the severity field and is not assigned/assignable.
The incident below was triggered on a service with alerts set up to trigger as suppressed by default.