Global Event Routing

What is Event Routing?

Routing enables you to send events to a single endpoint and write rules to define which team receives alerts based on content in those events.

API-based events with JSON/PD-CEF fields and email-based events are both supported.

Accessing, creating, and formatting routing rules

Access to create global event rules are granted to the account owner. With advanced permissions, global admins will also have access to view event rules.
To enable all users with view-only access to event rules, please contact PagerDuty Support.

Click on the Configuration menu at the top of the page on your PagerDuty account and choose Event Rules. Once on that page, click New Event Rule. You will be asked to fill in two sections: conditions and actions.

A condition indicates what you would like Event Routing to search for in inbound events. The corresponding action indicates what you would like Routing to do with an event that contains the selected condition.

You can either configure these in the UI, or use the Global Event Rules API.

Supported fields for writing rules

Events sent through the API

You can use the JSON field names directly, i.e. description. For nested fields, separate names with a dot (.), i.e. payload.summary.

If you are sending data through additional fields, enter them exactly as they are sent to PagerDuty. For example, if your events have a tags field, enter that field name in your rule condition as tags.

Events sent through email

Rules may be based on the content of an email by entering the appropriate email field as the event field. The most common email fields are:

  • headers.from.0.address (the from address)
  • headers.subject (the subject line)
  • body (the email body)

If your monitoring tool sends emails with custom headers, you can create rules based on those headers by including the prefix headers in front of the email header name when you enter that field name in your rule condition.

Field selector

You can view and choose relevant fields from your most recent event by using our JSON field selector:

The JSON field selector will appear when you select View a recent API event... in the dropdown that appears when you click Enter field name while writing your rules.

Formatting rules

What operations are supported?

  • A field contains/does not contain a value.
  • A field equals/does not equal a value.
  • A field exists/does not exist.
  • A field matches/does not match a regular expression. Regular expressions must use RE2 syntax. Note that unlike service-level event rules, global rules only support one (e.g. the first) capture group within the regex.

If you are new to regex, we suggest testing your rules with an online regex tester. regex101 and rubular are both excellent tools.

How do I use negative operations?

Rules with negative operations, such as “does not contain” or “does not equal”, will match events when the field in question does not exist. As an example:

  • severity field does not equal critical

These rules will also match any events where the severity field does not exist. If you'd like to avoid this, you must add an additional condition that matches only when the field exists. For example:

When all conditions are true:

  • severity field exists
  • severity field does not equal critical

Note that you must select that all conditions must be true for the rule to match.

What actions are supported?

Action
Description

Route to a service

Send events to a service and follow the settings on that service for creating incidents and sending notifications to responders. For more information on defining how notifications will be sent to responders, see service urgencies.

Route to a service and suppress

No incident is created, no notifications are sent. Events can still be routed to a service, meaning they have the associated service name listed in the Alerts table.

Set severity

For events that do not have a severity level (such as emails), users can define one, or a new severity level can be set.

Severity can be used for Dynamic Notifications, which are defined by the service settings. To use Dynamic Notifications, the events must be routed to a service that sets Dynamic Notifications based on severity levels.

Do not route to a service

You may choose not to route events to a service. In this case, your events are suppressed and will not be connected to a service. This is because the rules that determine incident settings (such as escalation policy assignment and notification settings) are associated with a service, and non-routed events do not have a service associated with them.

Where do I send events to use routing?

The Event Rules page (Configuration Event Rules) will display your routing integration key, as well as an endpoint and email address that you can use for routing. This integration key is case-sensitive.

You can use the V1 or V2 events endpoint as well.

If using the V1 events API, use your routing integration key as your service_key value in your event JSON.

If using the V2 events API, use your routing integration key as your routing_key value in your event JSON.

Troubleshooting event routing:

If none of your rules are working

This may be because you have not set up the integration key correctly in your monitoring tool. For event rules to work, you must use the integration key found to the right of your rules list under Integration Settings. If you are sending events through the Events API, use this integration key in the service_key field (Events API V1) or the routing_key field (Events API V2).

If one of your rules is not working

This may be because you have not entered the field name correctly in the rule conditions. The field name is based on the data that is sent to PagerDuty, and is not necessarily the same as the field name visible in the alert or incident in PagerDuty. Refer to the data in your monitoring tool or integration for the correct field name.

For nested fields, format the field name with periods separating each level. For example, field_group.sub_field.another_sub_field.

If you have confirmed your field names are set up correctly, you may have a rule that is too broad that is ordered before the rule that is not working. Re-order your rules so that more specific rules are listed first.

You may also want to check for rules with negative conditions ("does not contain" or "does not equal"), as these rules can match many events that do not contain the specified field. (See How do I use negative operations?)

Adding Notes with Event Rules

Notes can be used to help responders resolve incidents quicker by including information or links related to the system that the event comes from.

Notes via Event Rules functionality is part of our Event Intelligence product, which is purchased separately from the core PagerDuty platform.

Notes are added to incidents via the Global Event Rules Engine workflows. Users first set specific criteria in which notes information will be added. Users can then detail which information or links they want to include in the note.

Notes are added to incidents via the Global Event Rules Engine workflows. Users first set specific criteria in which notes information will be added. Users can then detail which information or links they want to include in the note.

If the event is added to an incident, notes can be seen on that incident's details page. Notes added by event rules will list the first account user as the author.

Scheduled Rules

You can gain a greater degree of control over your event rules by detailing specific times in the future in which they will be active. This can be useful for testing rules and for planned maintenance.

Scheduled Rules functionality is part of our Event Intelligence product, which is purchased separately from the core PagerDuty platform.

Scheduled Rules are set within Global Event Rules Engine workflows. Users first set specific time window in which the rules they have created will apply.

Users can then specify what actions are to be performed within this time window. For instance, users might want to suppress notifications.

In this example, if alerts come into PagerDuty from host123, they will only be suppressed if they occur within the 5 minute window immediately following the creation of the rule. After 5 minutes, the rule will become inactive, and an alert from host123 will create an incident in PagerDuty.

Threshold Alerts

Receive PagerDuty notifications only when your customized alert conditions breach your specified limits. In this way, responders effectively reduce alert noise without missing critical issues.

The Threshold Alerts functionality is part of our Event Intelligence product, which is purchased separately from the core PagerDuty platform.

Thresholds are set within Global Event Rules Engine workflows. Users first set specific rule criteria, which, if met, will be included in their threshold count.

Users can then select a service to route it to in the Alert Behavior section.

Once the service is selected, you will see an Incident Behavior section appear below. Here you can configure threshold alerting.

In this example, if five alerts come in from host123 within a 10 minute time window, the threshold will be exceeded and a PagerDuty incident will be created.

Thresholds + Alert Grouping

PagerDuty’s Alert Grouping feature can be used in combination with Threshold Alerting in order to reduce the number of incidents created, minimize noise, and keep the responder focused on the issue. While Threshold Alerting determines how many alerts are required to create an incident and notify the on-call user, Alert Grouping can take this a step further by then grouping any alerts that exceed this threshold into a single, open incident.

The current alert grouping options that are included with the Event Intelligence package are Time-Based Alert Grouping and Intelligent Alert Grouping.

Users can view all alerts, including those that do not breach the set thresholds, in the Alerts table. Alerts not breaching the thresholds will appear in the table with a status of Triggered (Suppressed). When the threshold is breached, the alert that exceeded the threshold will appear with a Triggered status.

This alert will also create an incident, which can be viewed in the Incidents Table.

Q&A:

Are email integration filters/rules, or event transformers supported?

For events routed by the routing engine, service-level logic applies. Event transformers are currently not supported.

Are events transformed into our Common Event Format (CEF)?

Events that are routed are transformed into CEF. Custom Event Transforms (CETs) are not supported.

What happens to events that don’t match any rules?

Currently, any events that are sent through global event rules and do not match any rules are suppressed. You can find them in the alerts table. You can change this default behavior by editing the catch-all rule at the bottom of your rules list.

Is there a way to bulk add/delete/edit these Global Routing rules?

Yes. You can leverage the Global Event Rules API to create a script which will allow this functionality.