Event Orchestration

Event Orchestration allows users to route events to an endpoint and create nested rules, which define sets of actions to take based on event content. With Event Orchestration, PagerDuty’s event rule system has been modernized and substantially enhanced to tackle your existing and future automation goals.

🚧

Required User Permissions

The following user roles can create/edit/delete Event Orchestrations:

  • User

  • Admin

  • Manager base roles and team roles. Manager team roles can create/edit/delete Event Orchestrations associated with their team.

  • Global Admin

  • Account Owner

📘

What’s the difference between Event Rules and Event Orchestrations?

Many of the new features in Event Orchestration are enhancements on event rules, however, there are also some substantial departures from existing event rule functionality:

  • With Event Orchestration, rules are now evaluated iteratively, starting with routing and ending with the definition of incident creation behavior at the service level. This is substantially different from the former approach where Global Event Rules (GERs) and Service Event Rules (SERs) are evaluated at the same time. This change has allowed us to remove both GERs and SERs from Event Orchestration.

  • As an intentional corollary of this change, Event Orchestration now also supports evaluation of conditions against either Raw or CEF formatted event payloads.

  • Another substantial change with Event Orchestration is the introduction of different rule types, as well as a rigid rule structuring. This was done to improve observability, ensure context is established when actions are taken, and to streamline rule creation.

Configure Event Orchestration

Step 1: Create an Orchestration

There are two types of Orchestrations:

Global Orchestrations

When an incoming event stream has more than one service destination, you can use Global Orchestrations to route events from the same source to different services.

To create a new Global Orchestration:

  1. Navigate to Automation Event Orchestration and select + New Orchestration.
  2. Enter an Orchestration Name and an optional Orchestration Description.
  3. Click Save. Continue to Step 2: Create Rules.

Service Orchestrations

When integrations exist on a service, you can use Service Orchestrations to evaluate your incoming events and perform additional actions.

To create a new Service Orchestration:

  1. Navigate to Services Service Directory click your desired service name and select the Settings tab.
  2. Scroll to Event Management and click the Service Orchestration Rules button. Click the + New Rule button and input an optional name or description for the Orchestration.
  3. Click Next. Continue to Step 2: Create Rules.

📘

Switch to Service Orchestrations

If you have a service that uses Service Event Rules, you can switch to Service Orchestrations at any time. Please read the section Switch to Service Orchestrations for more information.

Create a Service Orchestration from a Global Orchestration

A Service Orchestration can also be created after a Global Orchestration is made. This method allows users to create the rules that define what happens after events are routed to a service.

To create a new Service Orchestration from a Global Orchestration:

  1. Navigate to Automation Event Orchestration select your desired Global Orchestration and then click + New Service Route.
  2. Select a service from the Service dropdown.
  3. Select the type of condition needed to route events to the service.
  4. Click Save. Continue to Step 2: Create Rules.

Step 2: Create Rules

With Event Orchestration there are now two distinct types of rules that a user can create:

  • Routing Rules: Routing rules define how events are routed to services. They must be created in order for events to be routed to a service from a Global Orchestration.
  • Service Rules: Service rules are optional rules that can be created after a routing rule. They determine which actions are applied to events during processing, as well as how events eventually turn into incidents. Service rules will have their own conditions and will be linked to rules that precede and follow them.

Recommended Preparation

We recommend sending alerts to your Global Orchestration integration in order to inspect the incoming JSON key:value pairs and build rules from there.

To send alerts to Global Orchestrations:

  1. Navigate to Automation​​ Event Orchestration, select your preferred Orchestration and click the Global Orchestration Key dropdown.
  2. From this dropdown, copy the appropriate Integration Key to use in your upstream event source.

Routing Rules

  • Routing rules must be created in order for events to be routed to a service from a Global Orchestration. They are the first rule a user can create in an Event Orchestration and must be created before any other rule.
  • There can only be one routing rule per service per Event Orchestration. However, there could be other routing rules for a service in other Event Orchestrations.
  • If events are sent directly to a service, then the Service Orchestrations in place on the service will be evaluated. Global Routing Rules will not apply for Service Orchestrations.
  • Routing rule conditions can be based on either CEF transformed event data or raw event payloads.

Create a Routing Rule

To create a new routing rule:

  1. From an Event Orchestration page, click the + New Service Route button.
  2. Optionally enter a Description for your new rule.
  3. Next, indicate the type of conditions that should be met, based on your preferences.
  4. In the following condition section, indicate the event conditions that you would like the orchestration to match using one of the following methods:

Method

Description

Base conditions on incoming JSON

Depending on your account's activity, you may have recent events appear on the left side of the screen if events have been sent to the incoming event source. View these events to determine which values to use.

Events sent through the API

You will use the JSON field names directly (e.g., summary). For nested fields, separate names with a dot (.) (e.g., payload.taskid). If you are sending data through additional fields, enter them exactly as they are sent to PagerDuty. For example, if your events have a tags field, enter that field name in your rule condition as tags.

Events Sent Through Email

Note: Only available with Global Orchestrations, not with Service Orchestrations.

Rules may be based on the content of an email by entering the appropriate email field as custom details in the event field.

The most common email fields are:
-event.custom_details.from[0](the from address). *

  • event.custom_details.subject (the subject line)
  • event.custom_details.plain_body (the email body)

* The [0] refers to the 1st position in a list of emails. If you would like to generally search through a list of emails (either in the "to" field or the "from" field), please set to event.custom_details.from or event.custom_details.to.

  1. In the middle dropdown, select how the event should be filtered.

Filter Options

Description

  • matches part
  • does not match part

The field contains/does not contain a value.

  • matches
  • does not match

The field equals/does not equal a value (this operation requires the field to be passed in as a string).

  • exists
  • does not exist

The field exists/does not exist.

  • matches regex
  • does not match regex

The field matches/does not match a regular expression. Regular expressions must use RE2 syntax.

📘

Negative Operations

Rules with negative operations, such as does not contain or does not equal, will match events that do not contain your specified value and events that do not contain the field at all. As an example:

  • severity field does not equal critical

  • This will match events where the severity field does not equal critical and events that do not contain a severity field at all.

If you'd like to avoid this, you must add an additional condition that matches only when the field exists. For example:

When all conditions are true:

  • severity field exists

  • severity field does not equal critical

Note that you must select that all conditions must be true for the rule to match.

  1. In the second value field, input the value that should be met from the payload. This can be a string or regular expression.

📘

Case Sensitivity

Condition values in Event Orchestrations are case-insensitive.

For example, if a condition is set with Summary matches part DOWN, this will match if the Summary contains Down, down and other variations of the word.

  1. When additional conditions should be added, use the following options:
  • + And: Additional conditions should be met.
  • + New Condition: Create another set of conditions that should be met. A new condition block creates an OR operator of conditions that will also be evaluated alongside other blocks.
  1. Click Save to save the conditions.

📘

Note

There is no limit to the number of conditions that can be created per Orchestration.

Service Rules

  • Service rules are optional rules that can be created after a routing rule. These rules will have their own conditions and will be linked to rules that precede and follow them.
  • Service rules can include any action except those that define how an event is routed to a service.
  • When there is more than one potential service rule that an event could match against, that group of service rules is called a Set. Events will match against whatever service rule in a set has the highest ordinality (appears first, either in the UI or API).
  • When creating a service rule, any conditions can be created and combined, similar to a routing rule, however, threshold-based conditions can only ever exist on their own. If a user wants to create a rule with a threshold-based condition they must make a specific service rule that only has that condition present.

📘

Note

Service Rules can be used independently on Services. If using Service rules in combination with Global Orchestrations, a Routing Rule must be created before creating a Service Rule.

Configure a Service Rule

  1. Click + New Rule next to the condition block that should have additional actions applied.
  2. A modal will appear on the page. Select the type of condition, then specify the appropriate conditions for this new rule.
  3. Click Next
  4. Specify the actions that should be applied in the following menu:

Incident Actions

Determine how the incident should be created. Select whether you would like to Create an incident or Suppress.

Incident Action Option

Instructions and Details

Leave alert as-is

This triggers an incident on the account as soon as the event reaches PagerDuty.

Create an alert but pause notifications

An incident can trigger after a predetermined amount of time. Enter the amount of time you would like to pass before an incident and notifications are created in the Suspend alert for _ second(s) before triggering incident field. See Paused Incident Notifications for more details.

Suppress alert

Alerts that match the routing rule conditions will suppress, meaning they will not create PagerDuty incidents or notify responders. Suppress alerts are visible in the Alerts table.

Set Priority

To set priority, check the Set Priority checkbox and select your desired priority level from the dropdown.

Incident priority allows the classification of incidents based on a level of prioritization. Incident priority must be enabled on your account before it can be set with orchestrations.

Add a note to the incident

To add a note, under Additional Context, enter the text of the note that you would like to be added to an incident that meets your rule’s criteria.

Notes can be used to help responders resolve incidents quicker by including information or links related to the system that the event comes from.

Alert Actions

Alert Action

Instructions and Details

Set Severity

To set severity, check the Set Severity checkbox and select info, error, warning or critical from the dropdown.

All current pricing plans include incident severity.

Severity can be used for Dynamic Notifications, which are defined by the service settings. To use Dynamic Notifications, the events must be routed to a service that sets Dynamic Notifications based on severity levels.

With severity controlled via Event Orchestrations and the service configured to use dynamic notifications, you can control the incident urgency.

Set a custom trigger / resolve action

If you would like to automatically trigger an alert or resolve an alert based on your event rule conditions, check Set a custom trigger / resolve action and then select either Always trigger an alert or Always resolve an alert. All pricing plans include incident custom trigger/resolve actions with event rules.

Transformations

Dynamic Field Enrichment & Extraction is a tool to normalize event payloads using Event Rules. This capability allows you to copy important data from any combination of source event fields into any PagerDuty Common Event Format (CEF) field. Translate difficult machine terms and code into helpful context for responders so they can effectively respond to the problem. Dynamic Field Enrichment & Extraction can also be used to enhance Event Intelligence capabilities by customizing alerts, influencing the Intelligent Alert Grouping and Intelligent Triage machine learning algorithms.

Dynamic Field Enrichment & Extraction is composed of two functions:

  • Define Custom Variables: Capture snippets from a source event using regex matching for later use.
  • Replace Event Field: Enrich and update and CEF field or even create new key values within the Custom Details object using custom variables. When events are sent to PagerDuty, they are transformed into Common Event Format. Previously, Event Rules could only replace Summary or add a Deduplication Key, and these abilities were limited to a direct mapping from one field as a full replacement. Now, this ability is extended to replace any CEF field or even create new key values within the Custom Details object.

Dynamic Field Enrichment & Extraction is available with both Global Orchestrations and Service Orchestrations. For Service Orchestrations, custom variables must extract from CEF fields of the event payload, where for Global Orchestrations, custom variables can extract from any field of the event payload.

Create Custom Variables

Create Custom Variables

Instructions and Details

Add Variable

Click Add Variable to the right of Define Custom Variable.

Name

Enter a short descriptive Name to represent the value that you will use later on to define a CEF field.

Regex

Enter a Value for the variable.

Values are defined using valid RE2 regular expression syntax, and you can add as much complexity as you want with Regex to customize what part of each field you capture into a variable.

Source

Enter an origin Source.

You may use Sample Events on the right as a reference for variables by clicking Show Details. You may add multiple variables by clicking Add Variable in the upper right of the Customize Event Fields panel, or delete variables by clicking the trashcan icon directly to the right of the variable.

Replace Event Field

Replace Event Field

Instructions and Details

Common Event Field (CEF)

Select your preferred Common Event Field (CEF) from the left dropdown.

Right Dropdown

Select whether you would like to use Regex OR a Template from the right hand dropdown.

Regex option

If you chose the Regex option from the dropdown, enter a Value and the origin Source.

Template option

If you chose a text Template from the dropdown, you can reference a variable defined in the Create Custom Variables section (above) using {{ }} brackets (e.g. {{class}}) in the Value field.

Deduplication

The dedup_key field is used to merge events into a single alert. Events with the same dedup_key can update the status of the alert they are automatically merged into.

To set a deduplication key:

  1. In the Customize Event Fields section of your rule, under Define Custom Variable, create a variable from event fields by entering a Name, Value and the Source. You may use Sample Events on the right to create these variables.
  2. Next, under Replace Event Field, in the Event Field (CEF) field select Dedup Key from the dropdown. Select whether you would like to use Regex or a Template from the right hand dropdown. If you are using a Template, you can reference a variable defined in step 1 using {{variable name here}} brackets as the Value. If you are using Regex, enter a Value and the Source.

🚧

Using Dynamic Field Enrichment & Extraction for Email Events

Dynamic Field Enrichment & Extraction capabilities are available only for email events sent to Global Orchestrations. Users can extract from the email event using regex matching into multiple variables. Currently, only Summary and Dedup_Key Common Event Fields can be replaced for email events.

Process Automation

Process Automation allows users to specify which PagerDuty Automation Action should be run based on an incoming event’s payload. Responders can initiate automated diagnostics or remediation as soon as an incident is created, saving critical time during major incidents, or preventing major incidents altogether.

Event Orchestration is able to automatically trigger any Automation Action that has already been created and is allowed for use on the service Event Orchestration is being used on. Please read our Automation Actions article to learn more about how to configure actions and their permissions.

Automation Actions can only be enabled for Service Orchestration Rules.

📘

Requirements

This feature is only available for Business and Digital Operations plans with the following criteria:

Business Plans: Accounts must have the Event Intelligence and PagerDuty Automation Actions add-ons.
Digital Operations Plans: Accounts must have the PagerDuty Automation Actions add-on.

Please contact our Sales team if you are interested in Event Intelligence, and you may fill out this form if you are interested in Automation Actions.

To select an Automation Action:

  1. Select an Automation Action from the Automation Action dropdown. Initially this dropdown will say “No automation selected” until an Automation Action is specified.
  2. Only one Automation Action can be specified per rule, however multiple Automation Actions can be triggered during incident creation by nesting rules one after another.

📘

Note

If an Event Orchestration rule has an Automation Action specified that is no longer valid due to it being deleted or not available, the rule with the Automation Action will persist and still function, however the Automation Action will no longer trigger. Rules where this occurs will have a yellow icon on the Process Automation tab when you edit the rule, to indicate that the Automation Action is no longer available.

Webhooks

Webhook Actions allow response teams to easily define a dynamic webhook through an Orchestration and send a custom payload to a specified endpoint. This enables responders to automate actions like restarting a server, clearing logs, and reverting bad deploys.

The payload can be set using a combination of several predefined variables, created using Dynamic Field Enrichment and Extraction. Automated Triggers can be created using service orchestration rules.

Option

Instructions and Details

Enable webhook

Check the checkbox to enable and then select if the webhook should be manually or automatically triggered.

Name

Enter a button name.

API Endpoint

Enter the API endpoint for the payload.

Headers

Select the + Header Field button and enter the Name and Value for the webhook header.

Dynamic Field Enrichment and Extraction (Optional)

Button names and parameters can take advantage of the variables created using the Dynamic Field Enrichment and Extraction feature in the preceding Transformations section to further enrich the custom action.

  1. Click Save to save this new rule.

PagerDuty Condition Language (PCL)

If you need more control over the routing logic than the UI offers, you can write rules in PagerDuty Condition Language (PCL). To access this feature, select while creating a routing rule.

PagerDuty Condition Language (PCL)PagerDuty Condition Language (PCL)

PagerDuty Condition Language (PCL)

For more information, please see our developer documentation PCL Overview.

Copy Service Rules

📘

Copied Service Rule Limitations

  • Copied service orchestration rules will persist for 5 minutes after being copied or until they are manually deleted via the successful rule copy icon located in the top right of the service orchestration canvas.
  • Copying of Event Orchestration Routing Rules is not currently supported.

To copy an existing service orchestration rule:

  1. Click the menu shown and select Copy Rule.
  2. An icon will appear in the top right of the service orchestration canvas indicating that a rule has been copied when this is successful. Copying a rule copies all of its conditions, actions, and action configurations.
  3. Copied rules can be pasted after, before, or between any existing service orchestration rules. To paste a rule, you may either click the button shown on the service orchestration canvas between existing rules, or click the menu for an existing rule and select Paste Rule.
  4. A rule creation modal will appear and it will contain all the conditions, actions, and action configurations for the rule that was copied. These conditions, actions, and action configurations can be edited, or they can be left as they are. Click Save.

Edit Rules and Paths

Edit a Global Routing Rule

  1. Navigate to Automation and select Event Orchestration.
  2. Select the Orchestration from the list that contains the routing rule you wish to edit.
  3. Select the routing rule that you would like to edit.
  4. On the following page, click the button on the routing rule node, this will open a modal to change conditions.
  5. Make any changes to conditions or rule information and then click Save.

Edit a Service Rule on a Global Orchestration

  1. Navigate to Automation and select Event Orchestration.
  2. Select the Orchestration from the list that contains the routing and service rule you wish to edit.
  3. Select the routing rule that contains the service rule you would like to edit.
  4. On the following page, click the button on the service rule node, this will open a modal to change conditions.
  5. Make any changes to conditions or rule information and then click Save.

Edit Service Orchestration Rules

  1. Navigate to Services Service Directory and select your preferred service.
  2. Select the Settings tab, scroll to the Event Management section and click Service Orchestration Rules.
  3. On the following page, click the button on the service rule node, this will open a modal to change conditions.
  4. Make any changes to conditions or rule information and then click Save.

Delete a Global Routing Rule

  1. Navigate to Automation and select Event Orchestration.
  2. Select the Orchestration from the list that contains the routing rule you wish to delete.
  3. Select the routing rule you would like to edit from the list.
  4. Click the button on the routing rule, then select Delete Rule from the available options.
  5. Alternatively, select the routing rule, on the following page, click the to view available options.
  6. Select Delete.

Switch to Service Orchestrations

  • If your account has services that are currently using Service Event Rules, you may choose to switch to our updated Service Orchestrations. When you switch to Service Orchestrations, events that land on the service, regardless of source, would be evaluated by Event Orchestration instead of Event Rules.
    • Email events are the only exception to the above. If an email comes in on a service key, the Service Orchestration is ignored, and only the email filters and rules are applied. If an email comes in on a Global Orchestration, then the Global Routing Orchestration is applied, and if there’s a Service Orchestration, it is also applied.

To switch to Service Orchestrations:

  1. Navigate to Services Service Directory and select your desired service.
  2. Select the Settings tab, scroll to Event Management and then click Switch to Service Orchestrations.

The service can be switched back to processing events using Event Rules with the same process.

FAQ

How many services can an Event Orchestration route to?

You can route events to as many as 1,000 services in one orchestration. However, each service can only have one routing rule.

Can a service have more than one set of Service Orchestration rules?

No, each service can only have one set of Service Orchestration rules (orchestration rule canvas). Here, the same set of orchestration rules will be applied to all events that land on the service, which streamlines the Event Management process.

Can events be routed to a service from the Unrouted rule with Event Orchestration?

Yes, selecting a service to route events that do not match with any of the routing rules in an orchestration is possible with Event Orchestration.

Does Event Orchestration support email events?

Yes, Event Orchestration supports email events. Rules may be based on the content of an email by entering the email field as custom details in the event field event.custom_details.subject. Having email events as custom fields means email events are treated as CEF events that Dynamic Field Enrichment and Extraction can be applied to.

Does Event Orchestration support the "contains/does not contain" filter?

Yes, Event Orchestration supports the "contains/does not contain" filter. It has, however, been reworded and replaced with the "matches part/does not match part" filter.

Are condition values case sensitive with Event Orchestration?

No, condition values with Event Orchestration are case insensitive.

Does Event Orchestration support time-based conditions?

Yes, Event Orchestration supports time-based conditions but the functionality is different from time-based conditions in Event Rules. With Event Rules, time-based actions work as secondary conditions that must be met before actions are applied.

These conditions can be set up front with Event Orchestrations to avoid trapping events that don't match condition pairs. With Event Orchestration, events outside the set schedule would be evaluated by the next rule.

Is there a payload size limit for Event Orchestration paths?

Yes, the maximum payload size for an Event Orchestration is 8 MB. Orchestration paths greater than 8 MB will return 413 Payload Too Large errors and will not save.


Did this page help you?