PagerDuty Automation Actions
Run diagnostic or remediation actions at the push of a button
PagerDuty Automation Actions invoke jobs that are staged in Runbook Automation (SaaS or Self-Hosted). An Action can also invoke a script run by an installed Runbook Automation Self-Hosted Runner. By associating Automation Actions with a PagerDuty service, responders get push-button access to a library of defined diagnostic or remediation actions, resulting in shorter resolution times and fewer disruptive escalations.
Automation Actions can also be used as part of an Event Orchestration to enrich incident data with diagnostics information or perform remediation actions on incidents automatically.
Automated Diagnostics
Check out our Automated Diagnostics solution, which offers a prebuilt library of automation jobs that were specifically designed to be invoked using Automation Actions from within PagerDuty.
Availability
This feature is available to accounts on the following pricing plans: Business, Digital Operations (legacy), Enterprise for Incident Management, Customer Service Business, Customer Service Digital Operations (legacy) and Enterprise for Customer Service:
- Business Plans: Accounts must have the Event Intelligence and PagerDuty Automation Actions add-ons.
- Digital Operations (legacy) and Enterprise for Incident Management Plans: Accounts must have the PagerDuty Automation Actions add-on.
Please contact our Sales team if you are interested in PagerDuty AIOps, and you may fill out this form if you are interested in Automation Actions.
Configure Automation Actions
Runner Configuration
A Runner is the invocation method of an Automation Action. There are two types of Runners available: Runbook Automation Self-Hosted and Runbook Automation.
A software package must be downloaded, installed, and configured when configuring a Runbook Automation Self-Hosted Runner. This type of Runner is best paired with PagerDuty Runbook Automation Self-Hosted, but it can also be used to invoke a local script or a job in Rundeck OSS (version 4.0 and greater).
When configuring a Runbook Automation Runner, an API token for a valid Runbook Automation account is required to complete the setup wizard and create a link between a set of PagerDuty Incident Response and Runbook Automation accounts.
Required User Permissions
A PD Automation Actions user license is required for this feature. Please read Flexible Licensing for more information about assigning users a specific license type.
Users with the following roles can create, edit and delete Automation Actions and Runners:
- Account Owner
- Admin/Global Admin
Users with the following roles can create, edit and delete Automation Actions and Runners associated with their Teams:
- Manager Team roles
Users with the following roles can run Automation Actions on incidents:
- Account Owner
- Admin/Global Admin
- Manager base roles
- Responder
Users with the following roles can view all Automation Actions and Runners:
- Account Owner
- Admin/Global Admin
- Manager
- Responder
- Observer
There are two ways to create a Runner with PagerDuty:
- Create A Runbook Automation Self-Hosted Runner for our self-hosted automation platform.
- Create A Runbook Automation Runner for our SaaS-based workflow automation.
Create A Runbook Automation Self-Hosted Runner
Create a PagerDuty API Key
In the PagerDuty web app:
- Navigate to Integrations API Access Keys and click Create New API Key.
- Enter a Description that will help you identify the key later on. If you would like it to be read-only, check the Read-only option.
- Click Create Key.
- This will generate a unique key for the REST API. Copy it to a safe place, as you will not have access to copy this key again. Once it has been copied, click Close.
*If you lose a key, then you will need to delete it and create a new one.
Create a PagerDuty Runbook Automation Self-Hosted Token
In PagerDuty Runbook Automation Self-Hosted:
- Go to User Icon Profile.
- To the right of User API Tokens, click .
- Enter a name for the API token and click Generate New Token.
- Copy the User API Token and keep it in a safe place for later use.
Create a PagerDuty Runbook Automation Self-Hosted Project
- Create a Runbook Automation Self-Hosted Project that will contain the Job you’d like PagerDuty responders to be able to execute.
Create a PagerDuty Runbook Automation Self-Hosted Job
- Create a Runbook Automation Self-Hosted Job in the Project. After the job is created, copy its ID and keep it in a safe place for later use.
Create a Runbook Automation Self-Hosted Runner
Next, you will add a new runner to generate an ID and secret.
- In PagerDuty, navigate to Automation Automation Actions Runners tab Add Runner.
- Select the Process Automation runner type, and click Continue.
- Enter a Name and Description and click Continue.
- On the following screen, click Download ID & Secret to download the file
credentials.pdrunner-creds
, and click Confirm. - Open
credentials.pdrunner-creds
in your preferred text editor, replace<API_Token>
with the PagerDuty API key from step 8 above, and save the file:
id:XXXXXXXXXXXXXXXXXXXXXXX3P1C
secret:XXXXXXXXXXXXXXXXXXXX2I4B
token:<PagerDuty_API_TOKEN>
rundeck_url: http://localhost:4440
rundeck_token: <Your_Saved_PagerDuty_Process_Automation_API_TOKEN>
Security Tip
If you do not wish to store your runner credentials in plaintext, you can use environment variables instead. In zsh, for example, you can set can set an environment variable for the runner ID by adding
export RUNNER_ID=<runner-id-value>
to.zshenv
and then runningsource ~/.zshenv
to activate it.After setting and activating environment variables for
id
,secret
,token
,rundeck_url
andrundeck_token
, you can update the credentials file:id:${RUNNER_ID} secret:${RUNNER_SECRET} token:${RUNNER_PDTOKEN} rundeck_url:${RUNNER_CLOUD_RDKCLIENT_URL} rundeck_token:${RUNNER_CLOUD_RDKCLIENT_TOKEN}
-
Run the following commands in order to install Java 11+ on the machine that will host the runner. We recommend Linux (Ubuntu 18) or higher. If the operating system is reasonably new, we also recommend performing an update:
a. Update the Ubuntu version.sudo apt-get update
b. Install the open source Java version that the runner uses.
sudo apt install openjdk-11-jre-headless
c. Create a folder on your system in a preferred directory where you would like to place the credentials files.
mkdir automation_runner
d. Navigate to the runner’s directory.
cd automation_runner
e. Move the file
credentials.pdrunner-creds
from step 14 above into theautomation_runner
folder you just created. It should already contain the PagerDuty API key from step 15.f. Rename the file to
pdrunner-creds
.mv credentials.pdrunner-creds .pdrunner-creds
g. Download the latest runner software from PagerDuty.
wget https://runbook-actions.pagerduty.com/pd-runner.jar
h. View the current files in the directory:
.pdrunner-creds
andpd-runner.jar
.ls -larths
i. Launch
pd-runner
.java -jar pd-runner.jar
j. (Optional) Verify activity in
runner.log
.tail -f /home/ubuntu/automation_runner/runner/logs/runner.log
- Check the runner’s status in PagerDuty: Automation Automation Actions Runners. A green check mark indicates that the runner is active and running successfully. A red circle indicates that the runner is not running or there is a problem with the runner. Please verify your Java 11+ installation and check the local runner logs, or refer to our Automation Actions FAQ.
Create an Automation Action with a Runbook Automation Self-Hosted Runner
Once the Script & On Prem runner is running successfully, you can add an action.
- In PagerDuty, navigate to Automation Automation Actions Actions tab and click Add Action.
- Select a Script & On Prem runner, and click Continue.
- Perform the following:
Field | Instructions |
---|---|
Name the action | Enter a Name for the action. |
Describe the action | Describe the purpose of the action. |
Select Type | Select script OR process automation to determine how you would like the action to be invoked. |
Type of action | Select Diagnostic OR Remediation. |
Define your action | Depending on your Select Type selection above, enter the following: - Enter your desired script: This can be a script in a particular language, or a simple Unix command. The only requirement is that the runner host is capable of running the script and supports the script’s language.OR - Enter Process Automation Job ID: Enter your desired Process Automation Job ID. This should be an existing job ID in the Process Automation instance. (Optional): It is also possible to enter Process Automation arguments. For example, passing the PagerDuty incident ID at runtime as a string argument: -pd_incident_id ${pagerduty.incidentId} |
Link the runner that will run this action on your infrastructure | Select the runner from the dropdown menu. |
Associate this action with one or more services. The action will be available on any incidents that trigger on the selected service(s) | Select the PagerDuty service from the dropdown menu. |
Associate with a Team to limit who has access to run this action | Associate this action with a particular PagerDuty Team (Optional). |
Create A Runbook Automation Runner
Create a PagerDuty Runbook Automation Token
In Runbook Automation (Rundeck):
- Go to User Icon Profile.
- To the right of User API Tokens, click .
- Enter a name for the API token and click Generate New Token
- Copy the User API Token and keep it in a safe place for later use.
Create a PagerDuty Runbook Automation Project
- Create a Runbook Automation Project that will contain the Job you’d like PagerDuty responders to be able to execute.
Create a PagerDuty Runbook Automation Job
- Create a Runbook Automation Job in the Project. After the job is created, copy its ID and keep it in a safe place for later use.
Create a Runbook Automation Runner
- In PagerDuty, navigate to Automation Automation Actions Runners tab Add Runner.
- Select the Runbook Automation runner type, and click Continue.
- Enter a Name and Description and click Continue.
- On the following screen, enter your Runbook Automation instance’s subdomain and Runbook Automation User API key from step 4 above, and click Confirm.
- The confirmation page will present a green check mark indicating that the runner is active and running successfully.
Create an Automation Action with a Runbook Automation Runner
Once the Runbook Automation runner is running successfully, you can add an action.
- In PagerDuty, navigate to Automation Automation Actions Actions tab and click Add Action.
- Select a Runbook Automation runner, and click Continue.
- Select a Runbook Automation Job that is available for the Runbook Automation runner from the list.
- Perform the following:
Field | Instructions |
---|---|
Name the action | Enter a Name for the action. |
Describe the action | Describe the purpose of the action. |
Associate this action with one or more services. The action will be available on any incidents that trigger on the selected service(s) | Select the PagerDuty service from the dropdown menu. |
Associate this action with a Team to limit who has access to run this action | (Optional) Associate this action with a particular PagerDuty Team. |
- Select Continue to complete the configuration. You may repeat step 12 to create more actions.
Context Variables
Use Context Variables in Automation Actions
You can enrich Runbook Automation Self-Hosted, Runbook Automation, and Script actions with context variables that use data from the first triggering alert attached to an incident. Where possible, you should send event data to PagerDuty in our Common Event Format (PD-CEF), since using standardized event data will make it easier to reuse actions for multiple event types.
You can reference alert fields using ${pd.alert.key}
syntax, where key
is the path of the JSON key name (for example, ${pd.alert.details.target.name}
). We also support square bracket notation for cases where event data may contain arrays or fields with dots in the name (for example, ${pd.alert.details.nodes[0].hostname}
).
Context Variable Placeholders
The following placeholders are available:
${incident.id}
${incident.first_alert_id}
${user.id}
${pd.alert.client}
${pd.alert.severity}
${pd.alert.dedup_key}
${pd.alert.event_class}
${pd.alert.source_component}
${pd.alert.source_group}
${pd.alert.source_origin}
${pd.alert.description}
${pd.alert.creation_time}
Incident Metadata
Above, you’ll notice two incident metadata context variables, ${incident.id}
, and ${user.id}
. These context variables represent the obfuscated incident ID of the incident and the user ID that invoked the Automation Action (i.e., the user ID of the person that pushed the button in the web or mobile app).
${pd.incident.id}
${user.id}
Examples
PagerDuty Automation Action (Script)
![Script with context variables Script with context variables](https://files.readme.io/27b2c75-context-variables1.png)
Script with context variables
PagerDuty Automation Action (Runbook Automation Self-Hosted Job)
![Process Automation with context variable Process Automation with context variable](https://files.readme.io/93daeec-process_automation_context_variable.png)
Process Automation with context variable
PagerDuty Automation Action (Runbook Automation Job)
![Runbook Automation with context variable Runbook Automation with context variable](https://files.readme.io/70b31ec-context-variables2.png)
Runbook Automation with context variable
Search and Filter
You can search and filter any Automation Actions and Runners that have already been created. Navigate to Automation Automation Actions and use the following features in their respective tabs:
Actions Tab
- Search: You may search by action name.
- Filters:
- Services
- Teams
- Type
- Category
Runners Tab
- Search: You may search by runner name or description.
- Filter:
- Status
Configure Automation Actions Log
The Automation Actions Log reduces the time responders need to diagnose an issue and plan their next actions by providing a summarized list of diagnostics and remediation results that were triggered during the incident.
To use the Automation Actions Log, you must first add the PagerDuty Incident Output log filter at the global or step level of a given job in Process or Runbook Automation. This log filter can capture a regex capture group or a configurable number of lines of the logs generated (last 5, 10, 25, or all log lines). A custom text annotation can be added to each output.
To enable credentials for Incident Output to be sent successfully back to PagerDuty Incident Response:
- Configure the PagerDuty plugin at either the system or project level with a global PagerDuty API key and a valid PagerDuty user email.
![Configure PagerDuty plugin](https://files.readme.io/41f9210-configure_PagerDuty_plugin.png)
Configure PagerDuty plugin
- When creating or editing a job, enable Send Incident Output to PagerDuty on the Execution Plugins tab.
![Send Incident Output to PagerDuty](https://files.readme.io/cea80f9-Send_Incident_Output_to_PagerDuty.png)
Send Incident Output to PagerDuty
- You can add Incident Output log filters at a job's global or step level. Multiple filters can be configured at each level. Each filter configured creates a separate entry in the action log view on an incident. The filter is evaluated for every step/node combination run during a given job.
![Select log filter](https://files.readme.io/852de0f-select_log_filter.png)
Select log filter
![Configure incident output log filter](https://files.readme.io/b9d6dc6-Configure_Incident_Output_Log_Filter.png)
Configure incident output log filter
- After an Automation Action has been triggered, data returned from each Incident Output filter can be viewed in the Automation Actions Log tab on an incident. Each Automation Action run creates a parent record that can be expanded to view the details of the workflow node/step data collected from the completed job.
![View Automation Actions Log output](https://files.readme.io/2dfba55-View_Automation_Actions_Log_Output.png)
View Automation Actions Log output
Delete an Automation Action
- Go to Automation Automation Actions.
- With the Actions tab selected, select the name of your desired action to go to its detail page.
- In the top right, click Delete Action.
- In the confirmation dialog, click Yes, delete.
Run an Action on an Incident
Once you have configured configured Automation Actions, PagerDuty responders can run them on incidents. Automation Actions can run either a provided script or trigger a Job in PagerDuty Process or Runbook Automation. Results of actions can be viewed on the Automation Actions Log tab.
Run an Automation Action in the Web App
- Navigate to an open incident on a service where the Automation Action is configured, click Run Actions and select your desired action.
- Click Run Job (or Run Script) when prompted.
![Run a script action](https://files.readme.io/2bf7bc6-run_a_script_action.png)
Run a Script action
Run an Action in the Mobile App
- Tap an incident to view its details page, then tap Automation Actions. Alternatively, you may also tap More Automation Actions.
- Select your preferred Automation Action from the list, then tap Run Script or Run Job, depending on the type of action that was selected.
- The Output tab will reveal the output of the script that was run or the job that was selected.
- The incident will also have a history of all Automation Actions that have been run on the incident in the Initiated Automation Actions section. Tap View [x] Actions and then tap an action to see its details.
Run an Action in Slack
You can execute a PagerDuty Automation Action from Slack, provided the following prerequisites are fulfilled:
- You have configured the Slack V2 Next Generation integration
- An active Runbook Automation Self-Hosted Runner
- In Slack, trigger an incident with the
/pd trigger
command.- Read more about PagerDuty slash command in Slack
- Select More actions… Run an action.
- If prompted, please follow the steps to authorize your user in Slack.
![Select Run an action Select Run an action](https://files.readme.io/50330c7-image2.png)
Select Run an action
- Select an action from the dropdown.
- Review the action’s details and click Run.
A message posts in Slack when an action begins, as well as when it finishes. Select View output report in the completion message for full details about the action.
![Action details in Slack Action details in Slack](https://files.readme.io/5d25c26-image4.png)
Action details in Slack
Run an Action in Incident Workflows
You can select Automation Actions as a step in Incident Workflows. Read Incident Workflows for more information.
![Run an Automation Action in Incident Workflows](https://files.readme.io/d07f952-run_an_action_in_incident_workflows.png)
Run an Automation Action in Incident Workflows
View Automation Action Output
You can view logs captured from Runbook Automation (SaaS and Self-Hosted) in the Automation Actions Log tab on an incident's details page. Each entry in the table corresponds to an Automation Action or Runbook Automation Job. You can expand the entry to see the job's targeted nodes and the steps executed against each node.
- Navigate to an incident's details page and select Automation Actions Log tab to see in progress and completed actions results.
- Access the job's output through the View output report link on each action result, or in the incident timeline's record of the completed action.
![View output report](https://files.readme.io/f12fdf7-view_output_report.png)
View output report
- (Optional): To see the full Runbook Automation (SaaS or Self-Hosted) job details, click View full output on the Output Report page.
![View full output](https://files.readme.io/1a79c8d-view_full_output.png)
View full output
![Process automation details in the incident timeline Process automation details in the incident timeline](https://files.readme.io/8276699-automation-actions-incident-timeline.png)
Automation details in the incident timeline
![Output tab Output tab](https://files.readme.io/cc1de7c-automation-actions-job-output-report.png)
Output tab
- Click View full output in Process Automation to view the Runbook Automation Self-Hosted Job and see the results.
![Full output Full output](https://files.readme.io/bfbf44c-rundeck-actions-script-rd-output.png)
Full output
- Select the Script tab on the output report to see the detailed view of the Process or Runbook Automation Job that ran.
![Script tab Script tab](https://files.readme.io/8f375a0-automation-actions-output-report-script.png)
Script tab
- The Script and Context tabs detail the settings and alert context captured at runtime of the action.
![Output tab Output tab](https://files.readme.io/d3f66e6-Screenshot_2022-05-31_at_14.21.30.png)
Output tab
![Script tab Script tab](https://files.readme.io/c18224d-Screenshot_2022-05-31_at_14.24.00.png)
Script tab
![Context tab Context tab](https://files.readme.io/9773c69-context_tab.png)
Context tab
PagerDuty Advance for Automation Digest
Availability
PagerDuty Advance for Automation Digest is available in US and EU service regions.
PagerDuty Advance for Automation Digest synthesizes log output generated by Automation Action invocations and automatically generates a digest which provides:
- Incident details
- Action details
- Log summary
- Suggested next steps
To generate a PagerDuty Advance for Automation Digest:
- Navigate to your desired incident and scroll to the Automation Actions Log tab.
- Click Generate in the upper right to generate the Automation Digest.
FAQ
How does a Runbook Automation Runner connect to my Runbook Automation Account?
After successfully configuring a Runbook Automation Runner, Automation Actions uses the provided Runbook Automation API token to communicate directly with the API of a given Runbook Automation instance
What network port does a runner use?
The runner requires outbound access to PagerDuty’s public API: api.pagerduty.com
. There is no inbound access requirement unless you are using a Runbook Automation Self-Hosted Runner on a different machine than your Runbook Automation Self-Hosted instance. If this is the case, the runner requires inbound access to port 80 so that it can deliver its HTTPS requests to run jobs on your Runbook Automation Self-Hosted instance.
What happens if I trigger an action and the runner is down?
The action will be queued until the runner is available again. Once the runner is online, the pending action will execute.
How long does it take for an inactive runner to switch to offline (red status)?
A runner will switch to red status after 5 minutes of inactivity.
What should the successful messages look like in the runner.log
?
runner.log
?See output below:
ubuntu@ip-172-31-18-60:~$ more /home/ubuntu/automation_runner/runner/logs/runner.log
10:33:16.643 [main] INFO io.micronaut.context.env.DefaultEnvironment - Established active environments: [ec2, cloud]
10:33:18.389 [main] INFO com.dtolabs.automation.core.plugins.JarPluginScanner - Deleting plugin jar cache at /home/ubuntu/runner/runner/var/tmp/pluginJars
10:33:18.389 [main] INFO com.dtolabs.automation.core.plugins.JarPluginScanner - Deleting plugin lib dependency directory at /home/ubuntu/runner/runner/plugin-cache
10:33:18.450 [main] INFO io.micronaut.flyway.AbstractFlywayMigration - Running migrations for database with qualifier [default]
10:33:18.452 [main] INFO org.flywaydb.core.internal.license.VersionPrinter - Flyway Community Edition 7.5.3 by Redgate
10:33:18.621 [main] INFO org.flywaydb.core.internal.database.base.DatabaseType - Database: jdbc:sqlite:/home/ubuntu/runner/runner/storage/db.sqlite (SQLite 3.32)
10:33:18.657 [main] INFO org.flywaydb.core.internal.command.DbValidate - Successfully validated 1 migration (execution time 00:00.011s)
10:33:18.664 [main] INFO org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory - Creating Schema History table "main"."flyway_schema_history" ...
10:33:18.699 [main] INFO org.flywaydb.core.internal.command.DbMigrate - Current version of schema "main": << Empty Schema >>
10:33:18.702 [main] INFO org.flywaydb.core.internal.command.DbMigrate - Migrating schema "main" to version "1 - create tables"
10:33:18.713 [main] INFO org.flywaydb.core.internal.command.DbMigrate - Successfully applied 1 migration to schema "main" (execution time 00:00.015s)
10:33:19.210 [main] INFO io.micronaut.runtime.Micronaut - Startup completed in 2652ms. Server Running: http://localhost:4441
Is the runner process running on Ubuntu?
The command ps aux | grep -i runner
should output something similar to the following:
ubuntu 19741 0.7 9.6 3758548 386980 pts/0 Sl+ 10:33 0:55 java -jar pd-runner.jar
ubuntu 20385 0.0 0.0 14860 1144 pts/2 S+ 12:35 0:00 grep --color=auto -i runner
Is it possible to reveal the PagerDuty incident ID or the user who ran the Runbook Action from within a script?
Yes, you can access the incident ID with ${incident.id}
and the user ID with ${user.id}
. The example below shows how to escape them so that they print. These placeholders can also be used in a non-escaped fashion.
# script
df -h
echo "script ran by user:" ${user.id}
echo "for incident id" ${incident.id}
How can I set up the Automation Action with a Runbook Automation Self-Hosted Job that requires passing a PagerDuty incident ID as a variable?
This is possible by specifying an argument string when configuring an Automation Action. The string argument to be passed in is: -pd_incident_id ${pagerduty.incidentId}
.
Is it possible to disable the execution of Script Actions?
Yes, this is possible. By default, a Runner executes both Script and Runbook Automation Self-Hosted Job Actions. If execution of Script Actions is undesired, a Runner can be configured to do so via one of the following methods:
a) Add an override directive to the Runner configuration file: .pdrunner-config
script_type_actions_enabled: false
b) Supply a system property when starting a Runner
java -jar -Drunner.cloud.agentScriptOperations.enabled=false pd-runner.jar
c) To verify when Script Actions execution is disabled, the runner/logs/operations.log
will show a line similar to:
13:55:37.023 [main] INFO com.rundeck.sidecar.agent.operations.OperationService - Runner started. Version: 0.1.28 Operations registered: [RundeckCommand, TriggerRundeckJob, CancelInvocation, Ping, ExecuteAgentScript (disabled via configuration), RundeckFileCopy]
Note the ExecuteAgentScript (disabled via configuration)
statement confirms that execution of Script Actions has been disabled.
What is the minimum version of Rundeck OSS that can integrate with Automation Actions?
Automation Actions will work with Rundeck OSS version 4.0 and greater.
Updated 26 days ago