AWS CloudTrail + PagerDuty Benefits

• The AWS CloudTrail integration enables automated response workflows for activities that could affect compliance and security issues in your AWS ecosystem.
• PagerDuty provides the opportunity to group alerts alongside other ongoing issues and it has seamless integrations with systems of record like Jira and ServiceNow.

Requirements

To Configure the Integration:

• In PagerDuty: Managers, Admins, Global Admins and Account Owners can configure the integration.
• In AWS: Users who set up and manage AWS Config must have full-access permissions. Please read Amazon’s documentation What Is AWS Config? for more information about managing permissions in AWS.

How it Works

• When a CloudTrail metric goes beyond a predefined ALARM state threshold, a CloudWatch alert sends an event to a PagerDuty endpoint, triggering an incident.
• When the CloudTrail metric returns to an OK state below the predefined threshold, a resolve event is sent to the same endpoint, resolving the PagerDuty incident.

Version

This guide details configuration of the AWS CloudTrail V1 integration.

Integration Walkthrough

In PagerDuty

There are three ways to integrate AWS CloudTrail with PagerDuty:

• Integrate With Event Orchestration: Integrating with Event Orchestration may be beneficial if you want to build nested rules based on the payload coming from AWS.
• Integrate With a PagerDuty Service: Integrating with a PagerDuty service directly can be beneficial if you don’t need to route alerts from AWS to different responders based on the event payload. You can still use service-level Event Orchestration to perform actions such as alert suppression.
• Integrate With Event Rules: Integrating with global or service-level event rules may be beneficial if you want to build different rules based on the payload coming from AWS.

Integrate With Event Orchestration

Integrate With a PagerDuty Service

Integrate With Rulesets

In the AWS Management Console

1. In the Services search bar, search and select Simple Notification Service. In the SNS dashboard left menu, select Topics and click Create Topic on the right. This topic will be used to route alerts to PagerDuty from AWS.
2. Select the Standard Topic Type.
3. Next, perform the following:
   • Name: Enter a name for your topic. You may want to name your topic after your PagerDuty service’s name.
   • Display name (optional): Enter an optional display name.
   • Click Create topic at the bottom of the screen.
4. Now that your topic has been created, select Subscriptions in the left menu and click Create subscription.
5. Perform the following:
   • Topic ARN: Select the Topic ARN of the topic you just created.
   • Protocol: Select HTTPS.
   • Endpoint: Paste your Integration URL (generated in steps above).
   • Ensure that the Enable raw message delivery checkbox is unchecked.
   • Click Create subscription.
6. Your subscription should be automatically confirmed. Refresh the page to make sure the Status is Confirmed and not PendingConfirmation.
7. Next, search and select CloudTrail in the Services search bar.
8. Perform the following where necessary:

• If you have not already created a trail: Follow Amazon’s documentation to create a trail, ensuring that CloudWatch logs are Enabled during configuration. Note that you can optionally send the trail data to a newly created S3 bucket.
• If you have an existing trail and have not enabled CloudWatch logs: Follow Amazon’s documentation to enable CloudWatch logs in step 4.

9. After configuring CloudTrail to send data to CloudWatch Logs, navigate to the CloudWatch console to configure an alarm based on a CloudTrail finding. Navigate to Logs Log groups in the left menu.
10. Select the Name of the log group and then click Actions Create metric filter.
11. Enter the Filter pattern to use. For more information, see Amazon’s documentation on filter and pattern syntax. Click Next to continue.
12. On the next page, perform the following:

• Filter name: Enter a filter name.
• Metric namespace: Enter a name for the CloudWatch namespace where the metric will be published. If the namespace doesn't already exist, toggle Create new to the on position.
• Metric name: Enter a name for the new metric.
• Metric value: If your metric filter is counting occurrences of the keywords in the filter, enter 1. This increments the metric by 1 for each log event that includes one of the keywords.
  • Alternatively, enter a token such as $size. This increments the metric by the value of the number in the size field for every log event that contains a size field.
• Unit (optional): Select a unit to assign to the metric. If you do not specify a unit, the unit is set as None.
• Click Next to continue.

13. On the final page, review your metric filter and then click Create metric filter.
14. After creating the filter, you will see it listed in the Metric filters tab. Select the checkbox on your filter and then click Create alarm.
15. A new tab will open where you can configure a CloudWatch alarm. On the Specify metric and conditions
    screen, verify your alarm threshold and settings and then click Next to continue.
16. First, you will configure the In alarm state notification, which will trigger a PagerDuty incident when the metric has met your predefined threshold. Select the In alarm and Select an existing SNS topic radio buttons, and then select the SNS Topic (created above) from the Send a notification to… field.
17. On the next page, enter an Alarm name and click Next.
18. Next, you will configure the OK state notification, which will automatically resolve the PagerDuty incident if the metric has fallen back into an OK state (not meeting or exceeding the threshold). Click Configure actions in the left pane. Select the OK and Select an existing SNS topic radio buttons, and then select the SNS Topic (created above) from the Send a notification to… field. Click Next to continue.
19. An Alarm name and Alarm description from a previous step should be prepopulated. Click Next to continue.
20. On the Preview and create screen, review your alarm’s details. If you need to edit any details, click Edit to the right of each step. Once you have confirmed all details, click Create alarm. You should now see PagerDuty incidents when alarm thresholds are breached.