AWS CloudTrail Integration Guide | PagerDuty
Configure the AWS CloudTrail integration
AWS CloudTrail + PagerDuty Benefits
- The AWS CloudTrail integration enables automated response workflows for activities that could affect compliance and security issues in your AWS ecosystem.
- PagerDuty provides the opportunity to group alerts alongside other ongoing issues and it has seamless integrations with systems of record like Jira and ServiceNow.
Requirements
To Configure the Integration:
- In PagerDuty: Managers, Admins, Global Admins and Account Owners can configure the integration.
- In AWS: Users who set up and manage AWS Config must have full-access permissions. Please read Amazon’s documentation Granting Permissions for AWS Config Administration for more information about managing permissions in AWS.
How it Works
- When a CloudTrail metric goes beyond a predefined ALARM state threshold, a CloudWatch alert sends an event to a PagerDuty endpoint, triggering an incident.
- When the CloudTrail metric returns to an OK state below the predefined threshold, a resolve event is sent to the same endpoint, resolving the PagerDuty incident.
Version
This guide details configuration of the AWS CloudTrail V1 integration.
Integration Walkthrough
In PagerDuty
There are three ways to integrate AWS CloudTrail with PagerDuty:
- Integrate With Event Orchestration: Integrating with Event Orchestration may be beneficial if you want to build nested rules based on the payload coming from AWS.
- Integrate With Event Rules: Integrating with global or service-level event rules may be beneficial if you want to build different rules based on the payload coming from AWS.
- Integrate With a PagerDuty Service: Integrating with a PagerDuty service directly can be beneficial if you don’t need to route alerts from AWS to different responders based on the event payload. You can still use service-level Event Orchestration to perform actions such as suppressing.
Integrate With Event Orchestration
Configure a Global Orchestration Integration
- Configure a Global Orchestration in your PagerDuty account.
- Navigate to AIOps Event Orchestration click the name of your Global Orchestration.
- Click the Global Orchestration Key dropdown and then copy the Integration Key.
- Once you have your Integration Key, the Integration URL will be:
https://events.pagerduty.com/x-ere/[YOUR_INTEGRATION_KEY_HERE]
You can now proceed to the In the AWS Management Console section below.
Configure a Service Orchestration Integration
- Configure a Service Orchestration in your PagerDuty account.
- Create a Generic Events API integration on the same service.
- Once complete, copy the Integration Key and paste it into the following URL:
https://events.pagerduty.com/x-ere/[YOUR_INTEGRATION_KEY_HERE]
You can now proceed to the In the AWS Management Console section below.
Integrate With Rulesets
Rulesets End-of-Life
Rulesets and Event Rules will end-of-life in 2024. We recommend using Event Orchestration instead, which offers new functionality, such as improved UI, rule creation, APIs and Terraform support, advanced conditions, and rule nesting.
Configure a Global Ruleset Integration
- In the web app, navigate to AIOps Event Rules and select your Default Global Ruleset.
- On the Event Rules screen, click the Incoming Event Source dropdown and copy your Integration Key.
- Once you have your Integration Key, the Integration URL will be:
https://events.pagerduty.com/x-ere/[YOUR_INTEGRATION_KEY_HERE]
You can now proceed to the In the AWS Management Console section below.
Configure a Service Event Rules Integration
To use service-level event rules:
- Configure service event rules on your preferred service.
- Create a Generic Events API integration on the same service.
- Once complete, copy the Integration Key and paste it into the following URL:
https://events.pagerduty.com/integration/[YOUR_INTEGRATION_KEY_HERE]/enqueue
You can now proceed to the In the AWS Management Console section below.
Integrate With a PagerDuty Service
Add to a New Service
- To add the integration to a new service, navigate to Services Service Directory and click New Service.
- Follow the prompts and configure the service to your preferences. On the Integrations screen, select AWS CloudTrail from the search bar dropdown.
- Once you are done entering your service settings, click Create Service.
- You will now be in the service’s Integrations tab. Find your integration in the list and click the to view and copy your Integration URL. Keep it in a safe place for later use.
- You can now proceed to the In the AWS Management Console section below.
Add to an Existing Service
- To add an integration to an existing service, go to Services Service Directory and select the service where you would like to configure the integration. Select the Integrations tab and click Add another integration.
- Select AWS CloudTrail from the search bar dropdown.
- Click Add. Find your integration in the list and click the to the right to view and copy your Integration URL. Keep it in a safe place for later use.
- You can now proceed to the In the AWS Management Console section below.
In the AWS Management Console
- In the Services search bar, search and select Simple Notification Service. In the SNS dashboard left menu, select Topics and click Create Topic on the right. This topic will be used to route alerts to PagerDuty from AWS.
- Select the Standard Topic Type.
- Next, perform the following:
- Name: Enter a name for your topic. You may want to name your topic after your PagerDuty service’s name.
- Display name (optional): Enter an optional display name.
- Click Create topic.
- Now that your topic has been created, select Subscriptions in the left menu and click Create Subscription.
- Perform the following:
- Topic ARN: Select the Topic ARN of the topic you just created.
- Protocol: Select HTTPS.
- Endpoint: Paste your Integration URL (generated in steps above).
- Ensure that the Enable raw message delivery checkbox is unchecked.
- Click Create Subscription.
- Your subscription should be automatically confirmed. Refresh the page to make sure the Status is
Confirmed
and notPendingConfirmation
. - Next, search and select CloudTrail in the Services search bar.
- Perform the following where necessary:
- If you have not already created a trail: Follow Amazon’s documentation to create a trail, ensuring that CloudWatch logs are Enabled during configuration. Note that you can optionally send the trail data to a newly created S3 bucket.
- If you have an existing trail and have not enabled CloudWatch logs: Follow Amazon’s documentation to enable CloudWatch logs in step 4.
- After configuring CloudTrail to send data to CloudWatch Logs, navigate to the CloudWatch console to configure an alarm based on a CloudTrail finding. Navigate to Logs Log groups in the left menu.
- Select the Name of the log group and then click Actions Create metric filter.
- Enter the Filter pattern to use. For more information, see Amazon’s documentation on filter and pattern syntax. Click Next to continue.
- On the next page, perform the following:
- Filter name: Enter a filter name.
- Metric namespace: Enter a name for the CloudWatch namespace where the metric will be published. If the namespace doesn't already exist, toggle Create new to the on position.
- Metric name: Enter a name for the new metric.
- Metric value: If your metric filter is counting occurrences of the keywords in the filter, enter 1. This increments the metric by 1 for each log event that includes one of the keywords.
- Alternatively, enter a token such as $size. This increments the metric by the value of the number in the size field for every log event that contains a size field.
- Unit (optional): Select a unit to assign to the metric. If you do not specify a unit, the unit is set as None.
- Click Next to continue.
- On the final page, review your metric filter and then click Create metric filter.
- After creating the Filter, you will see it listed in the Metric filters tab. Select the checkbox on your filter and then click Create Alarm.
- A new tab will open where you can configure a CloudWatch alarm. On the Specify metric and conditions
screen, verify your alarm threshold and settings and then click Next to continue. - First, you will configure the In alarm state notification, which will trigger a PagerDuty incident when the metric has met your predefined threshold. Select the In alarm and Select an existing SNS topic radio buttons, and then select the SNS Topic (created above) from the Send a notification to… field.
- Next, you will configure the OK state notification, which will automatically resolve the PagerDuty incident if the metric has fallen back into an OK state (not meeting or exceeding the threshold). Click Add Notification. Select the OK and Select an existing SNS topic radio buttons, and then select the SNS Topic (created above) from the Send a notification to… field. Click Next to continue.
- On the next page, enter an Alarm name and Alarm description. Click Next to continue.
- On the Preview and Create screen, review your alarm’s details. If you need to edit any details, click Edit to the right of each step. Once you have confirmed all details, click Create alarm. You should now see PagerDuty incidents if the alarm thresholds are breached.
Updated 22 days ago