Event Management

Reduce the Number of Notifications for the Same Incident/Event

Notification fatigue can be assuaged with the following actions:

Include an Incident Key in your API Call

If your service is set up as a Generic API service type and multiple incidents are triggering for the same issue your team will be notified for each duplicate incident. To group these incidents you will want to include incident_key in your parameters for triggering incidents.

PagerDuty de-duplicates incidents based on the incident_key parameter — this identifies the incident to which a trigger event should be applied. If there are no open (unresolved) incidents with this key, a new incident will be created.

If there is already an open incident with a matching key, this event will be appended to that incident's alert log as an additional Trigger log entry.

If the event key field isn't provided, PagerDuty will automatically open a new incident with a unique key.

Adjust your Email Management Settings

If you have an email integration you can change your incident creation settings so that incidents are only triggered under certain conditions.

There are four types of email management settings:

  • Open a new alert/incident for each trigger email: Each email sent to the service's email address will open a new incident.
  • Open a new alert/incident for each new trigger email subject: Incidents are de-duped based on the subject of the trigger emails. For example, if two emails with the same subject are sent to this service's email address, the first opens a new incident, and the second is appended to this incident.
  • Open a new alert/incident only if an open incident does not already exist: An email sent to the service's email address will only open a new incident if they service has no open incidents; otherwise, the email will be appended to the open incident.
  • Create and resolve alerts/incidents based on custom rules: Use regular expressions to parse incident triggers and resolves.

The last three incident creation settings are ones that allow alert/incident de-duplication and reduce the number of notifications being sent.

To change the incident creation settings for your email integration service:

  1. Go to Configuration → Services
  2. Click on the name of the service that houses the integration, then select the Integrations tab. You can edit the integration by selecting Edit from the settings cog, or from the integration details page accessible by clicking the name of the integration.
  1. Select the appropriate email management setting from the four options provided.
  2. Click Save changes.

Suppression and Event Rules

The Event Rules feature is available on our current Standard and Enterprise plans. Please contact our Sales Team if you would like to upgrade to a plan featuring Event Rules.

Event Rules define automated actions to take on alerts created by services, based on conditions that apply to information in the inbound events' payloads. Event Rules can perform such actions as setting the severity of the resulting alert, or automatically suppressing the alert altogether.

Event Rules can be used on services that use vendor-specific and API-based integrations. For email integrations, we recommend utilizing email management rules.

Configure Event Rules For a Service

To use event rules on a service, that service must first be set to create Alerts and Incidents for inbound events. To do this, go to Configuration → Services, edit the service and locate the Incident Behavior section at the bottom of the Edit Service page.

Event Rules then can be added from the individual service page; the Event Rules tab is located on the far right.

Event Rule Behavior

Rules are evaluated in the order they appear in the Event Rules configuration tab, top to bottom. The first matching rule is applied, and then execution stops. If no rule matches, the event is processed ordinarily, as if there were no rules configured.

The order of rules can be controlled by dragging and dropping the rules in the Event Rules page.

If an event rule is deleted while a suppressed alert is in a "triggered" state, further alerts with the same alert key will continue to deduplicate until the alert is resolved.

Setting Conditions for Event Rules

When creating a rule, first set the conditions under which the rule will apply. For example:

In the above example, the rule will apply to an inbound event if the value of its «Source» field is exactly prod05-nginx. This particular rule will cause the event to be suppressed, and the severity will be set to Error.

Event rule conditions can be configured for any of the following PagerDuty Common Event Format (PD-CEF) fields:

  • Class
  • Component
  • Group
  • Location
  • Severity
  • Source
  • Summary
  • Custom Details

Accessing Custom Details

In order to access nested components of the custom details property for a comparison in a condition, enter the full namespace path, in dot notation, from the root of the details property, into the Key name in details field. For example, if the details property is {"foo": {"bar1": {"baz": "thevaluehere"}, "bar2": 2}}, the value in the comparison for foo.bar1.baz would be thevaluehere.

Once you have selected a field to examine, select a condition, such as equals or contains, and (if applicable) enter a value to compare the field against.

In addition to basic comparison, regular expressions can be used. Note, PagerDuty uses the RE2 flavor of RegEx to evaluate expressions.

Suppression

Suppression, as opposed to setting the severity of alerts, allows you to send events to PagerDuty without triggering any notifications. Suppressed alerts are stored in PagerDuty and available for forensics, analysis, and context, but do not create incidents. Suppressed alerts can be viewed in the alerts list as well as the Infrastructure Health Application.

To suppress an alert if it matches a given set of conditions, select «Suppress» as the action to take when you are building your Event Rules.

Viewing Suppressed Alerts

Suppressed alerts are filtered out of the incidents dashboard by default, including the incidents page for the service on which the suppressed alerts were triggered. Moreover, because suppressed events do not trigger incidents, they will not be visible in the mobile app.

They can be viewed in the Alerts tab at the top of your screen, between Incidents and Configuration.

Here is an example of a suppressed event. It looks very similar to other alerts, but has "Suppressed" in the severity field and is not assigned/assignable.

The incident below was triggered on a service with alerts set up to trigger as suppressed by default.

Infrastructure Health Application

Suppressed alerts may also be viewed in the Infrastructure Health Application. Suppressed alerts will be shown in gray to differentiate them from the rest of your alerts and incidents.

Event Management