Amazon CloudWatch + PagerDuty Benefits

• Amazon CloudWatch provides monitoring for AWS resources and customer-run applications. The service can collect data, gain insight, and alert users to fix problems within applications and organizations.
• Amazon CloudWatch gives system-wide visibility into resource utilization, and notifications can be set for metrics that cross specified thresholds. These notifications can be automatically sent to PagerDuty, which reliably alerts the correct on-call responder through their preferred contact methods.

Requirements

General:

• This integration expects to find in the Message property a nested JSON-encoded object; if this is not received, no alert will trigger. If you have any questions or need any assistance, please contact our Support team.

To Configure the Integration:

• In PagerDuty: Managers, Admins, Global Admins and Account Owners can configure the integration.

📘

Note

This integration is available for Amazon CloudWatch on AWS Cloud or AWS Outposts.

How it Works

• When an AWS service metric goes beyond a predefined threshold, a CloudWatch alert sends a notification to a PagerDuty endpoint, triggering an incident.
• When the AWS service metric returns to an OK state below the predefined threshold, a resolve event is sent to the same endpoint, resolving the PagerDuty incident.

Version

This guide details configuration of the Amazon CloudWatch V1 integration.

Integration Walkthrough

In PagerDuty

There are three ways to integrate Amazon CloudWatch with PagerDuty:

• Integrate With Event Orchestration: Integrating with Event Orchestration may be beneficial if you want to build nested rules based on the payload coming from AWS.
• Integrate With Rulesets: Integrating with global or service-level event rules may be beneficial if you want to build different rules based on the payload coming from AWS.
• Integrate With a PagerDuty Service: Integrating with a PagerDuty service directly can be beneficial if you don’t need to route alerts from AWS to different responders based on the event payload. You can still use service-level Event Orchestration to perform actions such as suppressing.

Integrate With Event Orchestration

Integrate With Rulesets

Integrate With a PagerDuty Service

In the AWS Management Console

1. In the Services search bar, search and select Simple Notification Service. In the SNS dashboard left menu, select Topics and click Create Topic on the right. This topic will be used to route alerts to PagerDuty from AWS.

2. Select the Standard Topic Type.

3. Next, perform the following:

   • Name: Enter a name for your topic. You may want to name your topic after your PagerDuty service’s name.
   • Display name (optional): Enter an optional display name.
   • Click Create topic.

4. Now that your topic has been created, select Subscriptions in the left menu and click Create Subscription.

5. Perform the following:

   • Topic ARN: Select the Topic ARN of the topic you just created.
   • Protocol: Select HTTPS.
   • Endpoint: Paste your Integration URL (generated in steps above).
   • Ensure that the Enable raw message delivery checkbox is unchecked.
   • Click Create Subscription.

6. Your subscription should be automatically confirmed. Refresh the page to make sure the Status is Confirmed and not PendingConfirmation.

7. Next, you will create a CloudWatch alarm that will send notifications to your SNS topic when a metric falls outside of a predefined threshold.

8. In the Services search bar, search and select Cloudwatch. Select Alarms All Alarms and then click Create Alarm on the right.

9. Click Select metric. Select your metric using either of the following methods:

   • Select the service namespace that contains the metric. Continue selecting your preferred options, which will narrow down your choices until a list of metrics appears. Select the check box next to your desired Metric Name.
   • In the search field, enter the name of a metric, dimension, or resource ID and hit Enter. Then select your desired results and continue selecting your preferred options until a list of metrics appears. Select the check box next to your desired metric.

Read more about commonly used metrics.

10. Next, select the Graphed metrics tab. Under Statistic, select one of the statistics or predefined percentiles, or specify a custom percentile (for example, p95.45). Under Period, select the evaluation period for the alarm. Click Select metric to continue.
11. On the next page under Conditions, select from the following Threshold types:

Click Next to continue.

12. First, you will configure the In alarm state notification, which will trigger a PagerDuty incident when the metric has met your predefined threshold. Select the In alarm and Select an existing SNS topic radio buttons, and then select the SNS Topic (created above) from the Send a notification to… field.
13. Next, you will configure the OK state notification, which will automatically resolve the PagerDuty incident if the metric has fallen back into an OK state (not meeting or exceeding the threshold). Click Add Notification. Select the OK and Select an existing SNS topic radio buttons, and then select the SNS Topic (created above) from the Send a notification to… field. Click Next to continue.

14. On the next page, enter an Alarm name and Alarm description. Click Next to continue.
15. On the Preview and Create screen, review your alarm’s details. If you need to edit any details, click Edit to the right of each step. Once you have confirmed all details, click Create alarm.
16. You should then see a confirmation dialog that your alarm was saved successfully.
17. The integration is now complete. When your alarm threshold is met, it will trigger an incident in PagerDuty. Once that alarm is back in an OK state, the incident automatically resolves in PagerDuty.

Commonly Used Metrics

Metrics that are commonly used with the Amazon CloudWatch integration include, but are not limited to:

EC2

To use the CloudWatch integration with EC2 instance metrics, follow the instructions in the Integration Walkthrough and perform the following when you Create a CloudWatch Alarm:

1. In step 9, select EC2 Per instance metrics.
2. Check the checkbox next to the Instance Name with your preferred Metric Name on the right. Commonly used metrics are CPU Utilization and Status Check Failed. Please read AWS’ documentation for more information on EC2 metrics.
3. Continue with the instructions in steps 10-17.

S3 Storage Lens

To use the CloudWatch integration with S3 Storage Lens metrics, follow the instructions in the Integration Walkthrough and perform the following when you Create a CloudWatch Alarm:

1. In step 9, select S3 Storage Metrics.
2. Check the checkbox next to the BucketName with your preferred Metric Name on the right. Commonly used metrics are Incomplete Multipart Upload Storage Bytes, Unencrypted Storage Bytes and Non-Current Version Storage Bytes. Please read AWS’ documentation for more information on S3 Storage Lens metrics.
3. Continue with the instructions in steps 10-17.

EKS

To use the CloudWatch integration with EKS metrics, follow the instructions in the Integration Walkthrough and perform the following when you Create a CloudWatch Alarm:

1. In step 9, select EKS Container Insights.
2. Check the checkbox next to your preferred Metric Name on the right. Commonly used metrics are cluster_failed_node_count and node_cpu_utilization. Please read AWS’ documentation for more information on EKS metrics.
3. Continue with the instructions in steps 10-17.

FAQ

What alarm statuses affect PagerDuty incidents?

If I use an email integration, how can I verify my PagerDuty service’s email address?

How can I change how events from CloudWatch are deduplicated into PagerDuty?

Why are my CloudWatch events not triggering incidents in PagerDuty?