PagerDuty Automation Actions

Run diagnostic or remediation actions at the push of a button

PagerDuty Automation Actions invoke jobs that are staged in Runbook Automation or Process Automation. An Action can also invoke a script run by an installed Process Automation Runner. By associating Automation Actions with a PagerDuty service, responders get push-button access to a library of defined diagnostic or remediation actions, resulting in shorter resolution times and fewer disruptive escalations.

Automation Actions can also be used as part of an Event Orchestration to enrich incident data with diagnostics information or perform remediation actions on incidents automatically.

👍

Automated Diagnostics

Check out our Automated Diagnostics solution, which offers a prebuilt library of automation jobs that were specifically designed to be invoked using Automation Actions from within PagerDuty.

📘

Pricing Plan and Feature Trial

PagerDuty Automation Actions are available as an add-on for Business and Digital Operations pricing plans. Please contact us if you would like to upgrade your plan or trial PagerDuty Automation Actions.

🚧

Required User Permissions

Users with the following roles can create, edit and delete Automation Actions:

  • Account Owner
  • Admin/Global Admin

Users with the following roles can create, edit and delete Automation Actions associated with their Teams:

  • Manager Team roles

Users with the following roles can view all Automation Actions:

  • Account Owner
  • Admin/Global Admin
  • Manager
  • Responder
  • Observer

Users with the following roles can invoke Automation Actions on incidents:

  • Account Owner
  • Admin/Global Admin
  • Manager base roles
  • Responder
  • Observer

Configure Automation Actions

📘

Runner Configuration

A Runner is the invocation method of an Automation Action. There are two types of Runners available: Process Automation and Runbook Automation.

When configuring a Process Automation Runner, a software package must be downloaded, installed and configured. A Process Automation Runner is best paired with PagerDuty Process Automation, but it can also be used to invoke a local script or a job in Rundeck OSS.

When configuring a Runbook Automation Runner, an API token for a valid Runbook Automation account is required to complete the setup wizard and create a link between a set of PagerDuty Incident Response and Runbook Automation accounts.

🚧

Required User Permissions

Users with the following roles can create, edit and delete Automation Actions:

  • Account Owner
  • Admin/Global Admin

Users with the following roles can create, edit and delete Automation Actions associated with their Teams:

  • Manager Team roles

Users with the following roles can view all Automation Actions:

  • Account Owner
  • Admin/Global Admin
  • Manager
  • Responder
  • Observer

Users with the following roles can invoke Automation Actions on incidents:

  • Account Owner
  • Admin/Global Admin
  • Manager base roles
  • Responder
  • Observer

There are 2 ways to create a Runner with PagerDuty:

Create A Process Automation Runner

Create a PagerDuty API Key

In the PagerDuty web app:

  1. Navigate to Integrations API Access Keys and click Create New API Key.
  2. Enter a Description that will help you identify the key later on. If you would like it to be read-only, check the Read-only option.
  3. Click Create Key.
  4. This will generate a unique key for the REST API. Copy it to a safe place, as you will not have access to copy this key again. Once it has been copied, click Close.
    *If you lose a key, then you will need to delete it and create a new one.

Create a PagerDuty Process Automation Token

In PagerDuty Process Automation (PA):

  1. Go to User Icon Profile.
  2. To the right of User API Tokens, click .
  3. Enter a name for the API token and click Generate New Token.
  4. Copy the User API Token and keep it in a safe place for later use.

Create a PagerDuty Process Automation Project

  1. Create a Process Automation Project that will contain the Job you’d like PagerDuty responders to be able to execute.

Create a PagerDuty Process Automation Job

  1. Create a Process Automation Job in the Project. After the job is created, copy its ID and keep it in a safe place for later use.

Create a Process Automation Runner

Next, you will add a new runner to generate an ID and secret.

  1. In PagerDuty, navigate to Automation Automation Actions Runners tab Add Runner.
  2. Select the Process Automation runner type, and click Continue.
  3. Enter a Name and Description and click Continue.
  4. On the following screen, click Download ID & Secret to download the file credentials.pdrunner-creds, and click Confirm
  5. Open credentials.pdrunner-creds in your preferred text editor, replace <API_Token> with the PagerDuty API key from step 4 above, and save the file:
id:XXXXXXXXXXXXXXXXXXXXXXX3P1C
secret:XXXXXXXXXXXXXXXXXXXX2I4B
token:<PagerDuty_API_TOKEN>
rundeck_url: http://localhost:4440
rundeck_token: <Your_Saved_PagerDuty_Process_Automation_API_TOKEN>

📘

PagerDuty Process Automation

If you are configuring the runner to connect to PagerDuty Process Automation, you may also enter your rundeck_url and rundeck_token so that the runner is able to communicate with your Rundeck instance.

  1. Install Java 11+ on the machine that will host the runner. We recommend Linux (Ubuntu 18) or higher. If the operating system is reasonably new, it is recommended to perform an update:
    a. sudo apt-get update: Update the Ubuntu version.
    b. sudo apt install openjdk-11-jre-headless: Install the open source Java version that the runner uses.
    c. mkdir automation_runner: Create a folder in a preferred directory on your system to place the credentials files.
    d. cd automation_runner: Navigate to the runner’s directory.
    e. Insert the credentials file that was downloaded from the PagerDuty Runner menu. It should already contain the PagerDuty API key inside the file.
    f. mv credentials.pdrunner-creds .pdrunner-creds: Rename the file to pdrunner-creds.
    g. wget https://runbook-actions.pagerduty.com/pd-runner.jar: Download the latest runner software from PagerDuty.
    h. ls -larths: View current directory to see all of the files. You should see two files from step 13.
    i. java -jar pd-runner.jar: Launch pd-runner.
    j. tail -f /home/ubuntu/automation_runner/runner/logs/runner.log: Optionally verify activity in runner.log.
  2. Next, check the runner’s status in PagerDuty. A green check mark indicates that the runner is active and running successfully. A red circle indicates that the runner is not running or there is a problem with the runner. Please verify your Java 11+ installation and check the local runner logs, or refer to our Automation Actions FAQ.

Create an Automation Action with a Process Automation Runner

Once the Script & On Prem runner is running successfully, you can add an action.

  1. In PagerDuty, navigate to Automation Automation Actions Actions tab and click Add Action.
  2. Select a Script & On Prem runner, and click Continue.
  3. Perform the following:
FieldInstructions
Name the actionEnter a Name for the action.
Describe the actionDescribe the purpose of the action.
Select TypeSelect script OR process automation to determine how you would like the action to be invoked.
Type of actionSelect Diagnostic OR Remediation.
Define your actionDepending on your Select Type selection above, enter the following:

- Enter your desired script: This can be a script in a particular language, or a simple Unix command. The only requirement is that the runner host is capable of running the script and supports the script’s language.

OR

- Enter Process Automation Job ID: Enter your desired Process Automation Job ID. This should be an existing job ID in the Process Automation instance.
(Optional): It is also possible to enter Process Automation arguments. For example, passing the PagerDuty incident ID at runtime as a string argument: -pd_incident_id ${pagerduty.incidentId}
Link the runner that will run this action on your infrastructureSelect the runner from the dropdown menu.
Associate this action with one or more services. The action will be available on any incidents that trigger on the selected service(s)Select the PagerDuty service from the dropdown menu.
Associate with a Team to limit who has access to run this actionAssociate this action with a particular PagerDuty Team (Optional).

Create A Runbook Automation Runner

Create a PagerDuty Runbook Automation Token

  1. Go to User Icon Profile.
  2. To the right of User API Tokens, click .
  3. Enter a name for the API token and click Generate New Token
  4. Copy the User API Token and keep it in a safe place for later use.

Create a PagerDuty Runbook Automation Project

  1. Create a Runbook Automation Project that will contain the Job you’d like PagerDuty responders to be able to execute.

Create a PagerDuty Runbook Automation Job

  1. Create a Runbook Automation Job in the Project. After the job is created, copy its ID and keep it in a safe place for later use.

Create a Runbook Automation Runner

  1. In PagerDuty, navigate to Automation Automation Actions Runners tab Add Runner.
  2. Select the Runbook Automation runner type, and click Continue.
  3. Enter a Name and Description and click Continue.
  4. On the following screen, enter your Runbook Automation instance’s subdomain and Runbook Automation User API key from step 4 above, and click Confirm.
  5. The confirmation page will present a green check mark indicating that the runner is active and running successfully.

Create an Automation Action with a Runbook Automation Runner

Once the Runbook Automation runner is running successfully, you can add an action.

  1. In PagerDuty, navigate to Automation Automation Actions Actions tab and click Add Action.
  2. Select a Runbook Automation runner, and click Continue.
  3. Select a Runbook Automation Job that is available for the Runbook Automation runner from the list.
  4. Perform the following:
FieldInstructions
Name the actionEnter a Name for the action.
Describe the actionDescribe the purpose of the action.
Associate this action with one or more services. The action will be available on any incidents that trigger on the selected service(s)Select the PagerDuty service from the dropdown menu.
Associate this action with a Team to limit who has access to run this action(Optional) Associate this action with a particular PagerDuty Team.
  1. Select Continue to complete the configuration. You may repeat step 12 to create more actions.

Context Variables

Use Context Variables in Automation Actions

You can enrich Process Automation, Runbook Automation and Script actions with context variables that use data from the first triggering alert attached to an incident. Where possible, send event data to PagerDuty in our Common Event Format (PD-CEF), since using standardized event data will make it easier to re-use actions for multiple event types.

You can reference alert fields using ${pd.alert.key} syntax, where key is the path of the JSON key name (for example, ${pd.alert.details.target.name}). Square bracket notation is also supported for cases where event data may contain arrays or fields with dots in the name (for example, ${pd.alert.details.nodes[0].hostname}). You can also reference fields that are not in the PD-CEF format.

Context Variable Placeholders

PD-CEF Placeholders

The following placeholders, based on standard PD-CEF fields, are available:

${pd.alert.class}
${pd.alert.component}
${pd.alert.dedup_key}
${pd.alert.group}
${pd.alert.source}
${pd.alert.summary}
${pd.alert.timestamp}

Non-Standard Fields

If the event is not normalized to PD-CEF, any value in the alert data can be referenced using the same {$pd.alert.key} structure. For example:

${pd.alert.description}
${pd.alert.source_origin}

Incident Metadata

Two incident metadata context variables are also available. These context variables represent the obfuscated incident ID of the incident and the user ID that invoked the Automation Action (i.e., pushed the button in the web or mobile app):

${pd.incident.id}
${pd.user.id}

Examples

PagerDuty Automation Action (Script)

1870

Script with context variables

PagerDuty Automation Action (Process Automation Job)

1700

Process Automation with context variable

PagerDuty Automation Action (Runbook Automation Job)

1999

Runbook Automation with context variable

Search and Filter

You can search and filter any Automation Actions and Runners that have already been created. Navigate to Automation Automation Actions and use the following features in their respective tabs:

Actions Tab

  • Search: You may search by action name.
  • Filters:
    • Services
    • Teams
    • Type
    • Category

Runners Tab

  • Search: You may search by runner name or description.
  • Filter:
    • Status

Delete an Automation Action

  1. Go to Automation Automation Actions.
  2. With the Actions tab selected, select the name of your desired action to go to its detail page.
  3. In the top right, click Delete Action.
  4. In the confirmation dialog, click Yes, delete.

Run an Action on an Incident

Once Automation Actions have been configured, PagerDuty responders can run them on incidents. Automation Actions can run either a provided script or trigger a Job in PagerDuty Process Automation.

Run a Runbook Automation Job in the Web App

  1. Navigate to an open incident on a service where the Automation Action is configured, click Run Actions and select your desired action.
  2. Click Run Job when prompted.
  3. Select the incident’s Timeline tab to see execution of the action from above. Click the output report link to view detailed output of the action.
  4. The Output tab will show the output of the Runbook Automation Job that was selected.
  5. Click View full output in Runbook Automation to view the Runbook Automation Job and see the results.
1999

Full output

  1. Select the Job Details tab to get a link to the Runbook Automation Job that ran.

Run a Process Automation Job in the Web App

  1. Navigate to an open incident on a service where the Automation Action is configured to run, click Run Actions and select your desired action.
  2. Click Run Job on the prompt that appears.
  3. Select the incident’s Timeline tab to see execution of the action from above. Click the output report link to view detailed output of the action.
1600

Process automation details in the incident timeline

  1. The Output tab will show the output of the Process Automation Job that was selected.
1600

Output tab

  1. Click View full output in Process Automation to view the Process Automation Job and see the results.
1600

Full output

  1. Select the Script tab to see the detailed view of the Process Automation Job that ran.
1600

Script tab

Run a Script Action in the Web App

  1. In the PagerDuty web app, navigate to an open incident on a service where the Automation Action is configured to run. Click Run Actions and select your desired action.
1740

Run a script action

  1. Select the incident’s Timeline tab to see notes related to the execution of the action from the previous step. Click the output report link to view detailed output of the action.
1084

Script details in the incident timeline

  1. The Output tab will reveal the output of the script that was run.
2006

Output tab

  1. The Script tab displays a detailed view of the script that was run.
1716

Script tab

  1. The Context tab displays the alert data used at the time of action invocation.
1652

Context tab

Run an Action in the Mobile App

  1. Tap an incident to view its details page, then tap Automation Actions. Alternatively, you may also tap More Automation Actions.
  2. Select your preferred Automation Action from the list, then tap Run Script or Run Job, depending on the type of action that was selected.
  3. The Output tab will reveal the output of the script that was run or the job that was selected.
  4. The incident will also have a history of all Automation Actions that have been run on the incident in the Initiated Automation Actions section. Tap View [x] Actions and then tap an action to see its details.

Run an Action in Slack

You can execute a PagerDuty Automation Action from Slack, provided the following prerequisites are fulfilled:

  1. In Slack, trigger an incident with the /pd trigger command.
  2. Select More actions… Run an action.
    • If prompted, please follow the steps to authorize your user in Slack.
634

Select Run an action

  1. Select an action from the dropdown.
  2. Review the action’s details and click Run.

A message posts in Slack when an action begins, as well as when it finishes. Select View output report in the completion message for full details about the action.

706

Action details in Slack

FAQ

How does a Runbook Automation Runner connect to my Runbook Automation Account?

Expand

After successfully configuring a Runbook Automation Runner, Automation Actions uses the provided Runbook Automation API token to communicate directly with the API of a given Runbook Automation instance

What network port does a runner use?

Expand

The runner requires outbound access to PagerDuty’s public API: api.pagerduty.com. There is no inbound access requirement unless you are using a Process Automation Runner on a different machine than your Process Automation instance. If this is the case, the runner requires inbound access to port 80 so that it can deliver its HTTPS requests to run jobs on your Process Automation instance.

What happens if I trigger an action and the runner is down?

Expand

The action will be queued until the runner is available again. Once the runner is online, the pending action will execute.

How long does it take for an inactive runner to switch to offline (red status)?

Expand

A runner will switch to red status after 5 minutes of inactivity.

What should the successful messages look like in the runner.log?

Expand

See output below:

[email protected]:~$ more /home/ubuntu/automation_runner/runner/logs/runner.log 
10:33:16.643 [main] INFO  io.micronaut.context.env.DefaultEnvironment - Established active environments: [ec2, cloud]
10:33:18.389 [main] INFO  com.dtolabs.automation.core.plugins.JarPluginScanner - Deleting plugin jar cache at /home/ubuntu/runner/runner/var/tmp/pluginJars
10:33:18.389 [main] INFO  com.dtolabs.automation.core.plugins.JarPluginScanner - Deleting plugin lib dependency directory at /home/ubuntu/runner/runner/plugin-cache
10:33:18.450 [main] INFO  io.micronaut.flyway.AbstractFlywayMigration - Running migrations for database with qualifier [default]
10:33:18.452 [main] INFO  org.flywaydb.core.internal.license.VersionPrinter - Flyway Community Edition 7.5.3 by Redgate
10:33:18.621 [main] INFO  org.flywaydb.core.internal.database.base.DatabaseType - Database: jdbc:sqlite:/home/ubuntu/runner/runner/storage/db.sqlite (SQLite 3.32)
10:33:18.657 [main] INFO  org.flywaydb.core.internal.command.DbValidate - Successfully validated 1 migration (execution time 00:00.011s)
10:33:18.664 [main] INFO  org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory - Creating Schema History table "main"."flyway_schema_history" ...
10:33:18.699 [main] INFO  org.flywaydb.core.internal.command.DbMigrate - Current version of schema "main": << Empty Schema >>
10:33:18.702 [main] INFO  org.flywaydb.core.internal.command.DbMigrate - Migrating schema "main" to version "1 - create tables"
10:33:18.713 [main] INFO  org.flywaydb.core.internal.command.DbMigrate - Successfully applied 1 migration to schema "main" (execution time 00:00.015s)
10:33:19.210 [main] INFO  io.micronaut.runtime.Micronaut - Startup completed in 2652ms. Server Running: http://localhost:4441

Is the runner process running on Ubuntu?

Expand

The command ps aux | grep -i runner should output something similar to the following:

ubuntu   19741  0.7  9.6 3758548 386980 pts/0  Sl+  10:33   0:55 java -jar pd-runner.jar
ubuntu   20385  0.0  0.0  14860  1144 pts/2    S+   12:35   0:00 grep --color=auto -i runner

Is it possible to reveal the PagerDuty incident ID or the user who ran the Runbook Action from within a script?

Expand

Yes, you can access the incident ID with ${incident.id} and the user ID with ${user.id}. The example below shows how to escape them so that they print. These placeholders can also be used in a non-escaped fashion.

# script
df -h
echo "script ran by user:" ${user.id}
echo "for incident id" ${incident.id}

How can I set up the Automation Action with a Process Automation Job that requires passing a PagerDuty incident ID as a variable?

Expand

This is possible by specifying an argument string when configuring an Automation Action. The string argument to be passed in is: -pd_incident_id ${pagerduty.incidentId}.

Is it possible to disable the execution of Script Actions?

Expand

Yes, this is possible. By default, a Runner executes both Script and Process Automation Job Actions. If execution of Script Actions is undesired, a Runner can be configured to do so via one of the following methods:

a) Add an override directive to the Runner configuration file: .pdrunner-config

script_type_actions_enabled: false

b) Supply a system property when starting a Runner

java -jar -Drunner.cloud.agentScriptOperations.enabled=false pd-runner.jar

c) To verify when Script Actions execution is disabled, the runner/logs/operations.log will show a line similar to:

13:55:37.023 [main] INFO  com.rundeck.sidecar.agent.operations.OperationService - Runner started. Version: 0.1.28 Operations registered: [RundeckCommand, TriggerRundeckJob, CancelInvocation, Ping, ExecuteAgentScript (disabled via configuration), RundeckFileCopy]

Note the ExecuteAgentScript (disabled via configuration) statement confirms that execution of Script Actions has been disabled.