Knowledge Base
API DocsIntegrationsPagerDuty UniversityPagerDuty CommonsContact SupportMy Support Tickets

Incident Workflows

Create packaged incident response actions

Use Incident Workflows to build an automated response for every incident. For example, you can configure one workflow that automatically triggers for all P1 and P2 incidents, and a separate workflow that orchestrates a security incident response. Through highly-configurable Triggers and Actions, you can apply 'if-this-then-that' logic to orchestrate the right response for your teams, processes, and business. Streamline your incident management processes even more by taking advantage of Workflow Integrations.

📘

Availability

Incident Workflows are included with the following pricing plans: Business, Digital Operations (legacy) and Enterprise for Incident Management.

Customers on Professional pricing plans have access to Incident Workflows and can create one Incident Workflow. This includes the following triggers and actions:

Please contact our Sales team to upgrade to a pricing plan with Incident Workflows.

Incident Workflows diagram

Incident Workflow Components

Incident Workflows are comprised of three main components: triggers, actions, and Workflow Integrations. By customizing the right set of triggers and actions, you can adjust workflows for your particular use case.

Triggers

Triggers determine when Incident Workflows should be run, and you can add multiple triggers to each workflow to customize your preferred response. For example, you can use triggers to create a workflow that only runs for high-priority incidents. There are four types of triggers: Incident Type, Conditional, Manual and API.

Incident Type Triggers

Incident Type triggers will automatically start an incident workflow whenever you create or set an incident of the specified type. You can select one or more Incident Types to trigger the workflow. This trigger automatically applies to all services on your account.

🚧

Required User Permissions

You must have Admin or Account Owner permissions to create an Incident Type trigger.

📘

Availability

Accounts are currently limited to a maximum of 5 Incident Type triggers.

Conditional Triggers

Conditional triggers can start Incident Workflows whenever an incident is created, or only when specific conditions are met.

  • If you select When an incident is created, you are not required to add a condition. If you do add a condition, though, the condition will be evaluated when the incident is created.
  • If you select When conditions are met, the condition is evaluated any time there is a change to the incident. This means that a workflow could potentially run multiple times. For example, if the condition is priority matches P-1 and a user toggles the priority back and forth between P-1 and P-2, the workflow will trigger each time the incident's priority changes to P-1.

📘

Availability

Conditional Triggers are only available on the Business and Enterprise for Incident Management plans.

📘

Workflow Trigger Description

If you select When conditions are met, the condition is evaluated every time there is a change to the incident. The workflow will trigger if:

  • The condition evaluates to true when the incident is initially created.
  • The condition changes from false to true upon an update to the incident.

Once triggered, the workflow will not activate again on subsequent changes if the condition remains true. However, if the condition changes to false and then back to true, the workflow will trigger again. This allows the workflow to potentially run multiple times during the lifecycle of an incident, but only when the condition transitions from false to true.

Scenario Example

Consider a workflow with a conditional trigger set to activate when the priority is P1 or P2:

  1. Incident Creation: The incident’s priority is set to P1. The condition evaluates to true, triggering the workflow.
  2. Priority Change to P2: The condition remains true, so the workflow is not triggered again.
  3. Priority Change to P3: The condition evaluates to false, and the workflow is not triggered.
  4. Priority Change Back to P1: The condition changes from false to true, triggering the workflow once more.

When a conditional trigger is used, it will perform actions automatically. Some actions, such as adding responders, require a user association for the request. When this occurs, you will see the incident's assignee (or first assignee if there are multiple users assigned) logged as the user performing the action in the incident timeline.

The following conditional trigger fields are available to all users with Incident Workflows:

  • Priority
  • Status
  • Urgency

In addition to the fields above, users with the Enterprise plan for Incident Management have access to the fields listed below. Please contact our Sales Team if you'd like to upgrade to an account with these fields:

  • Custom Fields
  • Created At
  • Incident Key
  • Incident Number
  • Resolved At
  • Service ID
  • Title
  • Updated At

🚧

Conditional Trigger Limits

You are limited to 10 enabled Incident Workflows with conditional triggers that apply to the same technical service and a limit of five enabled Incident Workflows with conditional triggers that apply to all services on your account.

Please read the section Disable an Incident Workflow below for more information about enabling and disabling workflows.

Manual Triggers

Manual triggers let responders start the Incident Workflow directly from an incident. When a manual trigger is added, Incident Workflows can be triggered from the PagerDuty web app, mobile app, Slack, or Microsoft Teams.

📘

Availability

Manual Triggers are only available on the Business and Enterprise for Incident Management plans.

🚧

Required User Permissions

If you cannot view a specific Workflow, it's because you are not on the Team specified in the Workflow's permissions. Please contact a Global Admin or Account Owner in your organization to edit the manual Workflow permissions for you.

API Triggers

You can trigger workflows using the Incident Workflows REST API endpoint. One of the following user roles is required to trigger workflows via the API:

  • Manager base role
  • Admin
  • Account Owner

Integration Triggers

You can trigger Incident Workflows directly from our Slack, Microsoft Teams, ServiceNow, Zendesk, Salesforce and Jira Server integrations.

Applying Triggers to Services

The last step of creating a trigger is applying it to the right services. There are two ways to apply a trigger to services:

  • Apply a trigger to all services on the account.
  • Apply a trigger to specific services.

Once a trigger has been applied to a service and the workflow has been published and enabled, it can then be used for incidents on those services.

Actions

Every Incident Workflow is made up of a set of actions that define what the workflow will do. You can review a list of supported actions in our Workflow Actions library.

Workflow Integrations

Workflow Integrations facilitate the connection between Incident Workflows and external tools such as AWS, Azure Functions, Datadog, and Web API, enabling seamless authentication and data exchange. Please read Workflow Integrations to learn more about how to streamline your incident management processes.

Create an Incident Workflow

🚧

Required User Permissions

Account Owners and Global Admins can create and edit all Incident Workflows. Other users can create and edit workflows and triggers for services where they have Manager access.

📘

Maximum Workflow Amount

Each account can create a maximum of 2,000 Incident Workflows.

Create an Incident Workflow from a Template

Incident Workflow templates provide a helpful starting point so you don’t have to create a Workflow from scratch. With templates, you can see best practices and review common examples of Incident Workflow use cases.

  1. In the web app's top navigation bar, go to Automation Incident Workflows.
  2. Select the template you would like to use:
  • Major Incident
  • Collaboration
  • Update Stakeholders

You will be directed to a preview of the template.

Incident Workflow templates

Incident Workflow templates

  1. Click Use Template to create a workflow based on the template you selected.
  2. In the builder view, you can optionally select the Settings tab to modify the workflow Name, Description, and Who can edit this Workflow?. Click Save.
  3. With the Builder tab selected, triggers and actions that require updates are marked Needs attention.
Needs attention banner
  1. Click a trigger or action to review its configuration:
    1. For any triggers or actions you wish to remove from the workflow, click Remove.
    2. Click to add additional triggers or actions to the workflow.
    3. To left of a trigger or action, click and drag to reorder items as needed.
  2. When you are satisfied with the workflow's configuration, click Publish in the upper right and then Publish again in the confirmation modal.

Create an Incident Workflow from Scratch

  1. Select Automation Incident Workflows from the top navigation bar.
  2. Click Create Workflow.
  3. Enter a workflow Name and Description. Under Who can edit this Workflow? select one of the following:
  • All admins
  • Team managers on a specific team: Select the preferred Team from the dropdown.
  1. Click Create.

Add Triggers

  1. Click Add Trigger and select from the following:
Incident Type Trigger: Starts an Incident Workflow when an incident is created with or set to a specific type

a. In the Incident Types dropdown, select your preferred Incident Type(s). You may select one or multiple.
b. Click Save.

Conditional Trigger: Starts an Incident Workflow when an incident in an associated service is created or updated

a. In the section When should this Workflow start? , select When an incident is created or When conditions are met (default) from the dropdown. This selection determines if a Workflow should start on incident creation, or only when the conditions are met.

b. Select the icon to use the Visual Condition Editor, or the icon to use the PagerDuty Condition Language (PCL) editor.

c. Configure the field, operator, and value to define the condition that must be met in order for the trigger to run. The list of available operators will dynamically update depending on the field's data type. Optional: Click And to add an AND condition. To add an OR condition, click New Condition. Then repeat this step.

d. In the This trigger applies to dropdown, select whether you would like your condition(s) to apply to All services or Specific services. If you select Specific services, then select your preferred service(s).

e. Click Save.

Manual Trigger: Starts an Incident Workflow when manually updated

a. In the Who can run this workflow? dropdown, select whether you would like to grant permissions to All incident responders or Incident responders on a specific team. If you selected Incident responders on a specific team, then select your preferred Team.

b. In the This trigger applies to dropdown, select whether you would like your condition(s) to apply to All services or Specific services. If you select Specific services, then select your preferred service(s).

c. Click Save.

Add Actions

  1. Click Add Action and select an action. Please reference the full list of Incident Workflows actions. Note: Actions will trigger in the sequential order that they are added to the Workflow.
  2. Repeat steps 4-5 until all of the appropriate triggers and actions have been added to the Incident Workflow.
  3. Click Publish in the upper right-hand corner of the page, and then click Publish again in the modal to confirm. The Incident Workflow will now be published and enabled.

Incident Workflow Drafts

Until an Incident Workflow is published, it exists as a draft. Draft Incident Workflows will not run, even when they have fully configured triggers. If you leave the workflow builder before an Incident Workflow has been published, it will remain in a draft state.

When you edit a published Workflow it creates a draft version, where changes can be safely made without affecting the published version. You can only have one draft at a time. When you publish a draft, it will replace the previous live version.

Edit an Incident Workflow

🚧

Required User Permissions

Account Owners and Global Admins can edit all Incident Workflows. Other users can edit workflows and triggers for services where they have Manager access.

  1. To edit an Incident Workflow as a draft, navigate to Automation Incident Workflows.
  2. Click to the right of the Incident Workflow you would like to edit and select Open in Builder.
    • If the Incident Workflow being edited has already been published, your edits will be applied to a draft Incident Workflow. This allows you to make changes to published Incident Workflows without impacting the Workflow currently being used. When you’re ready to apply those changes, you can re-publish the Workflow to make those changes live.
  3. Make your edits to Triggers, Actions and/or Settings and then click Save.

Edit Workflow Permissions

Workflow permissions are set per-Workflow, and you can grant users access by their role or Team. A Team must be given explicit permission to view or edit a Workflow, and only one Team can have access at a time.

Users with edit access will also be able to run the Workflow via API. Because of how permissions are handled, when using the Team managers on a specific team setting, users on that Team with an Observer or Responder role can also run the Workflow via API, though they cannot edit it. Additionally, when an incident triggers a Workflow, any user added to that incident as a responder can also run the Workflow via API.

🚧

Required User Permissions

Account Owners and Global Admins can edit Workflow permissions for all users. Other users can edit Workflow permissions for services where they have Manager access.

  1. Navigate to Automation Workflows.

  2. Select to the right of the Workflow you wish to edit and then select Open in Builder.

  3. Select the Settings tab. Under Who can edit this Workflow?, select one of the following options:

  • All admins
  • Team managers on a specific team: Under the Select team dropdown, select your preferred Team.
  1. Click Save.

Edit Manual Workflow Permissions

To run a Workflow manually, the Workflow must have a manual trigger, from which you can configure permissions. Manual Workflow permissions are set per-trigger.

🚧

Required User Permissions

Account Owners and Global Admins can edit manual Workflow permissions for all users. Other users can edit manual Workflow permissions for services where they have Manager access.

  1. Navigate to Automation Incident Workflows.

  2. Select to the right of the workflow you wish to edit and select Open in Builder.

  3. Select the Manual Trigger. Under Who can run this workflow?, select from one of the following:

  • All incident responders
  • Incident responders on a specific team: Under the Select team dropdown, select your preferred Team.
  1. Click Save.

Disable an Incident Workflow

If you'd like to make a published workflow temporarily unavailable, you can disable it. Disabled workflows cannot run via the API or conditional triggers, and will not appear on incidents in the list of manually-triggered workflows.

  1. In the PagerDuty web app, navigate to Automation Incident Workflows.
  2. To the right of the workflow you wish to disable, click and select Disable Workflow.
  3. In the confirmation modal, click Disable.

📘

Re-enable a Workflow

You can re-enable a disabled workflow by following the instruction above and selecting Enable Workflow in step two.

Delete an Incident Workflow

  1. In the PagerDuty web app, navigate to Automation Incident Workflows.
  2. To the right of the workflow you wish to delete, click and select Delete Workflow.
  3. In the confirmation modal, click Delete.
    1. Note: This is a non-reversible action and cannot be undone.

Field References

You can use field references to dynamically insert incident data into your Incident Workflow actions. During configuration, you may add data such as incident.id, incident.status, incident.priority, etc., to an action's input parameters, resulting in the workflow pulling the corresponding incident’s data each time it runs. This feature allows you to customize and standardize the information, settings, and flow of every response. You can browse all available field references by typing {{ in the input field or by clicking the icon on the right side of the input field.

Insert field reference icon

Insert field reference icon

Examples

  • Add Users as Responders: When you select this action, you have the option to include a message to responders. You may want to include standard incident data in the Message field, such as {{incident.service.name}} and {{incident.priority}}, so that responders see the service name and priority in the responder request.
  • Send Status Update: When you select this action, you may want to include {{incident.title}} and {{incident.priority}} in the Message field so that stakeholders see the incident’s title and priority in their status updates.
  • Create a Slack Channel for an Incident: When you select this action, you have the option to set a Slack channel name. You may want to include the {{incident.id}} in the Desired Channel Name field to get a unique channel name every time.

Supported Fields

Incident Workflows support referencing fields in most action configuration fields. Field references use the syntax {{example}}, which are called “double curly braces,” or sometimes "double handlebars".

Field name{{field-name}}
incident.custom_fields.CUSTOM_FIELD_NAME (will vary based on the Custom Field's name){{incident.custom_fields.CUSTOM_FIELD_NAME}}
incident.id{{incident.id}}
incident.url{{incident.url}}
incident.title{{incident.title}}
incident.status{{incident.status}}
incident.service.id{{incident.service.id}}
incident.service.name{{incident.service.name}}
incident.urgency{{incident.urgency}}
incident.priority{{incident.priority}}
incident.created_at{{incident.created_at}}
incident.updated_at{{incident.updated_at}}
incident.resolved_at{{incident.resolved_at}}
incident.incident_key{{incident.incident_key}}
incident.incident_number{{incident.incident_number}}
incident.conference_bridge.conference_url{{incident.conference_bridge.conference_url}}
incident.conference_bridge.conference_number{{incident.conference_bridge.conference_number}}
current_date{{current_date}}
time_zone{{time_zone}}
workflow.id{{workflow.id}}
workflow.name{{workflow.name}}

Action-Specific Field References

Most actions produce outputs, which you can use as field references in later steps of the workflow. While configuring an Incident Workflows action, you can find compatible outputs under the Steps tab. Select your desired action and output. It will be formatted as {{steps['ACTION_NAME'].fields['FIELD_NAME']}}.

Please see the full Workflow Actions library to review each action's available outputs.

Default Action Names

PagerDuty generates a default name for an action, for example Create Slack Channel for an Incident, which you can reference in later steps by entering {{steps['Create a Slack Channel for an Incident'].fields['Channel Link']}}.

If needed, you can rename an action by clicking Rename to the right of an action's title. If you rename an action, please also update the string name in the steps object. For example, if you rename the action to Auto-generate Slack Channel, you can reference its channel link with the following snippet: {{steps['Auto-generate Slack Channel'].fields['Channel Link']}}.

Differentiating Outputs

There are two options to consider in terms of naming actions:

  • If you add action steps with the same name in a workflow, they appear as one set of field references in the Field Reference picker. This might be desirable so that a subsequent step can overwrite the outputs, which may later be referenced from another step.
  • If you want to differentiate the outputs from each of the individual steps so you can uniquely reference the output from a specific step, you will need to rename the actions so that they are different from each other.

Example of Actions with the Same Name:

  • A workflow includes two actions, both named Send Get Request followed by a Condition that will send a message if the Condition is met.

    Workflow Condition with Two Actions, Both Named `Send GET Request`

    Workflow Condition with Two Actions, Both Named Send GET Request

  • The action-specific field reference from both actions is steps['Send GET Request'].fields['Response Status Code']

  • In this example, the second Send Get Request action will override the value stored in steps['Send GET Request'].fields['Response Status Code'] and the Condition can only check against the result of the second action.

Example of Actions with Different Names:

  • By renaming the actions so that they are each unique, the Output Fields from each action will have different identifiers and will thus not overwrite each other.

    Workflow Condition with Actions Named `Send GET Request Service A` and `Send GET Request Service B`

    Workflow Condition with Actions Named Send GET Request Service A and Send GET Request Service B

  • In this updated workflow, the action-specific field references are now distinct, steps['Send GET Request Service A'].fields['Response Status Code'] and steps['Send GET Request Service B'].fields['Response Status Code'].

  • The Condition can now check that both GET requests return a 200 status code before sending a Slack Message.

📘

Instance vs. Step Scope

  • Instance scope are fields that are reference-able and exist independently from the configured steps. For an Incident Workflow, a great example of this is the incident field -- it's reference-able within the workflow, but was not output by any step in the workflow.
  • Step scope is a field scope that targets the step name, but not anything inherently unique about the step. This gives you control over:
    • Having a later step with the same name override the prior step’s outputs (by sharing the same scope), or:
    • Having a distinct scope and separately tracked outputs (by making the step names distinct, even if the action each step executes is the same).

Examples

The following examples represent common use cases you might consider incorporating into your own Incident Workflows.

Assign an Incident Role to the On-Call User

If your organization uses the Incident Roles feature, a common task would be assigning a role to the on-call user from a specific schedule.

  1. While creating or editing an Incident Workflow, select Get On-Call User from a Schedule from the list of actions and click Add Action.
  2. Select your desired Schedule from the dropdown and click Save.
  3. In a later step, select Assign an Incident Role action and click Add Action.
  4. Select an incident role from the Role dropdown.
  5. In the Assignee click {+} Steps Get On-Call User from a Schedule On Call User ID. This will populate the appropriate reference field's information, for example {{steps['Get On-Call User from a Schedule'].fields['On Call User ID']}}.
  6. Click Save.
Create a Slack Channel and Include its URL in a Status Update

You may wish to create a Slack channel for an incident at the beginning of a workflow, and then reference its Channel Link in a status update later in the same workflow.

  1. While creating or editing an Incident Workflow, select Create Slack Channel for the Incident from the list of actions and click Add Action.
  2. Enter your desired information (i.e., Workspace, Channel Name, etc.) and click Save.
  3. In a later step, select Send Status Update and click Add Action.
  4. Under Status Update template, select Custom Message.
  5. In the Message field, enter any text you'd like to include and leave the cursor where you'd like to insert the channel's URL. Then click {+} Steps Create a Slack Channel for the Incident Channel Link. This will populate the appropriate reference field's information, for example {{steps['Create a Slack Channel for the Incident'].fields['Channel Link']}}.
Action-specific field reference for Slack channel link

Action-specific field reference for Slack channel link

  1. Click Save.

Run a Workflow

There are three ways to run Workflows:

When You Create or Set an Incident Type

Incident Type triggers will automatically start an incident workflow whenever you create or set an incident of the specified type. To run the workflow automatically, the workflow must have an Incident Type trigger. This trigger automatically applies to all services on your account.

🚧

Required User Permissions

You must have Admin or Account Owner permissions to create an Incident Type trigger.

📘

Availability

Accounts are currently limited to a maximum of 5 Incident Type triggers.

Manually Run an Incident Workflow

🚧

Required User Permissions

  • A PD Automation Actions user license is required to manually trigger Incident Workflows that include PagerDuty Automation Actions. Any responder can start all other workflows from an incident.
  • If you cannot view a specific Workflow, it's because you are not on the Team specified in the Workflow's permissions. Please contact a Global Admin or Account Owner in your organization to edit the manual Workflow permissions for you.

Manually Run a Workflow via Web or Mobile App

  1. In the web or mobile app, click the incident's Title to view its details page, and then click Run a Workflow in the actions menu.
    • In the mobile app you may also tap More and select Run a Workflow.
  2. Select your desired Workflow.
  3. Click Run Workflow.

To configure a manual trigger, follow the instructions in Create an Incident Workflow and select Manual Trigger when you Add Triggers to the Workflow.

Manually Run a Workflow via Slack

Please see our Slack User Guide for instructions.

Manually Run a Workflow via Microsoft Teams

Please see our Microsoft Teams Integration Guide for instructions.

Run an Incident Workflow when a Condition is Met

Incident Workflows with conditional triggers will run automatically when their preconfigured conditions are met. To run a Workflow automatically, the Workflow must have a conditional trigger.

To configure a conditional trigger, follow the instructions in Create an Incident Workflow and select Conditional Trigger when you Add Triggers to the Workflow.

View Incident Workflow Audit Trail Reporting

Please read our Audit Trail Reporting article for more information.

Incident Workflow Actions

Please see our Workflow Actions library for documentation on Incident Workflow actions.

FAQ

Why can't I add a trigger to all services?

Users can only select services that they have edit permissions on. To apply the trigger to a service you don't have edit permissions on, you'll need to request edit permissions on that service.

What happens if I change the trigger in a draft?

Incident Workflows in a draft state will not trigger under any circumstance. No changes made in a draft will affect live, published Workflows or incidents. Changes will only take effect after publishing.

Why can't I see a specific Incident Workflow in the Run a Workflow dropdown menu?

To run a Workflow manually, the Workflow must have a manual trigger. From the trigger, you can configure which services to associate with the Workflow.

If a Workflow is not appearing from the Run a Workflow menu on the incident, it is because of one of the following reasons:

  1. The Workflow does not have a manual trigger.
  2. The service or incident is not associated with the Workflow.
  3. The Workflow is associated to the incident, but did not meet the trigger conditions and is not available.

Do Incident Workflow actions run sequentially?

Yes. The order in which you add actions to an Incident Workflow defines the sequence in which they execute. For example, if you subscribe stakeholders and then send a status update, those stakeholders will be subscribed to the incident before the status update is generated.

Can I configure Incident Workflows using Terraform or the REST API?

Yes. Please refer to our Terraform Provider for PagerDuty and Incident Workflows REST API documentation for more information.

How much data can an Incident Workflow action return?

Incident Workflow actions can return up to 5MB of data in a response. If an action exceeds this limit, an Incident Workflow completed with error log entry is added to the incident with the message Encountered a system error after the action completed.

Are there rate limits for Incident Workflows?

Your PagerDuty account can handle up to 5,000 incident events per minute across all services that are using conditional triggers. If your account exceeds this limit, incident events will be added to a queue and processed after all other incident events have completed.

Example 1: If your account has one service with 10 conditional triggers, the service can process up to 500 incident updates per minute without a delay.

Example 2: If your account has five services, each with 10 conditional triggers, each service can process up to 100 incident updates per minute. Alternatively, one service can process up to 500 incident updates per minute, depending on how the load is distributed.

Are there any restrictions on the Web API actions?

Incident Workflows Web API Actions are subject to an account-wide rate limit of 500 MBs per minute. If your account exceeds that limit, all workflow actions will be paused for the remainder of the minute. Triggers will continue to fire without limitation, but workflow actions will not execute until the minute has passed. Please take care with workflows that run often and send or receive large amounts of data, since that can result in slowing down other, potentially more urgent workflows.

Which IP addresses will HTTP requests come from?

Incident Workflow actions that make external HTTP requests, including Web API actions (e.g., Send GET Request, Send POST Request, etc.), will originate from the list of IP addresses in our developer documentation Webhook IPs.

Can I format URLs in Slack messages so they are more user-friendly?

Yes. All Slack incident actions with a Message field support mrkdown syntax. As an example, if you have a lengthy Google cMeet URL (e.g., https://g.co/meet/critical_scheduled_cluster_export_failed-at-20XX-XX-XX), which you'd like to format as Google Meet Link, you can use the following snippet: <https://g.co/meet/critical_scheduled_cluster_export_failed-at-20XX-XX-XX|Google Meet Link>.

For more information, please read Slack's documentation Formatting text for app surfaces.

What is the maximum number of Incident Workflows with conditional triggers that can be associated with a service or account?

You can create and enable up to 10 Incident Workflows with conditional triggers that apply to the same technical service.

You can create and enable up to five Incident Workflows with conditional triggers that apply to all services on the account.

If you reach the trigger limit, you will see an error message indicating this when attempting to save the workflow.

Updated about 15 hours ago

Learn more