Time-Based Alert Grouping
Automatically group incoming alerts into open incidents based on the time they are generated
Time-Based Alert Grouping will automatically add alerts to an open PagerDuty incident for a predetermined period. This can be helpful for services that generate many alerts. Grouped alerts mean fewer incidents and interruptions for responders, richer context for triggered incidents, and lower resolution times.
Availability
This feature is available with our PagerDuty AIOps add-on, or with Legacy Event Intelligence. If you would like to sign up for a trial of PagerDuty AIOps features, please read PagerDuty AIOps Trials.
Required User Permission
Users with the following roles can edit a service’s Alert Grouping settings:
- Account Owner
- Admin and Global Admin
- User
- Manager base role and team roles
- Manager team roles can only manage services associated with their team.
Enable Time-Based Alert Grouping
To enable Time-Based Alert Grouping:
- In the PagerDuty web app, navigate to Services Service Directory and select your preferred service’s name.
- Select the Settings tab and click New Grouping/Edit in the section Reduce Noise.
- Select Time only and a duration from the dropdown.
- Click Save.
Disable Time-Based Alert Grouping
To select a different grouping method, or to disable Alert Grouping all together, in the PagerDuty web app:
- Navigate to Services Service Directory and select the name of your desired service.
- Select the Settings tab and click Edit in the section Reduce Noise.
- In the bottom-left, click Delete. In the confirmation modal, click Delete again.
- Note: This action cannot be undone.
Time-Based Alert Grouping Behavior
When Time-Based Alert Grouping is enabled on a service, the first alert will create a new incident. Subsequent alerts will be added to that incident for the specified time period while the incident is open (i.e., triggered or acknowledged). If the incident is resolved within the selected time period, the timer will stop, and no subsequent alerts will be grouped into that incident.
The timer restarts when the next alert on that service comes in and triggers a new incident. If the incident is not resolved within the selected time period, the incident will remain open and the next alert will trigger a new incident. Any subsequent alerts will be grouped into the most recent incident.
If you choose until incident is resolved for the timer’s duration, the first alert on the service will trigger an incident and all subsequent alerts will be grouped into that incident until it is resolved, regardless of how much time passes.
Time-Based Alert Grouping Example
In this example, we have a Shopping Cart service that uses Time-Based Alert Grouping with a 1-hour time window.
Time | Example 1: Incident resolves within 45 minutes | Example 2: Incident does not resolve within 1 hour |
---|---|---|
10:00 a.m. | TRIGGER: An alert triggers and creates Incident 1. The 1-hour timer begins, set to group all new alerts on the Shopping Cart service into Incident 1 until 11:00 a.m. | TRIGGER: An alert triggers and creates Incident 1. The 1-hour timer begins, set to group all new alerts on the Shopping Cart service into Incident 1 until 11:00 a.m. |
10:07 a.m. | ACK: Responder acknowledges Incident 1. | ACK: Responders acknowledges Incident 1. |
10:15 a.m. | Five more alerts come in on Shopping Cart. Incident 1 now has 6 alerts, all triggered. | Five more alerts come in on Shopping Cart. Incident 1 now has 6 alerts, all triggered. |
10:45 a.m. | RESOLVE: The responder resolves Incident 1. New alerts on the Shopping Cart service will no longer be grouped into incident 1. | Incident 1 is still in the acknowledged state, grouping alerts. |
10:55 a.m. | TRIGGER: An alert comes in on Shopping Cart service, creates Incident 2. A new 1-hour timer begins, set to group all new alerts on the Shopping Cart service into Incident 2 until 11:55 a.m. | An alert comes in on Shopping Cart, is grouped into Incident 1. Incident 1 now has 7 alerts: 3 resolved, 4 triggered. |
11:00 a.m. | Incident 1 is resolved, Incident 2 is triggered with 1 alert. | Incident 1 is still acknowledged, the timer expires and PagerDuty will no longer actively group new alerts into Incident 1. Incident 1 is acknowledged with 7 alerts. |
11:10 a.m. | An alert comes in, is grouped into Incident 2. Incident 2 now has 2 alerts. | TRIGGER: An alert comes in, creates Incident 2. The The 1-hour timer begins, set to group all new alerts on the Shopping Cart service into Incident 2 until 12:10 a.m. |
FAQ
Can I move grouped alerts to another incident?
If Time-Based Alert Grouping adds an alert to an incident that would belong better as part of a different incident, there is an option to move the alert to another incident or to create a new incident. For more information, please read Move Alerts to Another Incident.
How does Time-Based Alert Grouping affect incident notifications?
While Time-Based Alert Grouping is designed to reduce notification noise, the specific number of times you are notified will be based on your notification rules. If a service has an acknowledgement timeout, for example, responders will receive notifications when the timeout expires.
How is Time-Based Alert Grouping different from alert deduplication?
Deduplication is configured on a per-integration basis, and often requires specific event information, like an incident key or an alert key. When you have multiple integrations on a service, or your monitoring tool generates a variety of alerts, it can become increasingly difficult to manage. Time-Based Alert Grouping allows you to group all incoming events on the service into the current incident for the specified time period, regardless of the monitoring tool (i.e., integration) that generates the original alert.
How is Time-Based Alert Grouping different from snoozing incidents?
Snoozing an incident keeps it from moving up the escalation chain and notifying users. Snoozing does not prevent new incidents from generating on a service.
Can I manually merge incidents?
Yes, Time-Based Alert Grouping does not affect your ability to manually merge incidents or move alerts from one incident to another. All grouped alerts will be merged into a single incident.
Can manually triggered incidents be grouped with Alert Grouping?
Manually triggered incidents do not create alerts, do not participate in Alert Grouping, and cannot be merged into other incidents.
Will incidents created by REST API be grouped by Time-Based Alert Grouping?
No, like manually created incidents, the Incidents REST API bypasses alerts and directly creates incidents. With this in mind, it is not affected by Time-Based Alert Grouping.
This also applies to any integrations that create incidents via REST API, for example ServiceNow integration.
Events sent via the Events API v2 will work with Time-Based Alert Grouping.
Is there a limit to how many alerts can group into a single incident?
Yes, PagerDuty incidents are limited to 1,000 alerts. After this limit is reached, a new incident will be created and subsequent alerts will continue to group into the most recent incident.
Updated 15 days ago