Notification Bundling

Follow

Notification bundling is PagerDuty's term for the incident summary we switch to when multiple incidents are triggered and assigned to a user.

PagerDuty's mission is to notify you to the incidents you need to know about, at the level you need to know them. This directive becomes complicated when your notification volume explodes: if 100 notifications a minute are incoming, what is the best way to notify you?

Normally, we would notify you for each individual incident. However, if your notification method is set to phone, and your phone starts ringing off the hook until 100 consecutive phone calls and voicemails are pushed through, the problem becomes serious. We call these "notification storms," and calibrate our policies to avoid this.

Notification bundling means we roll together or bundle these multiple notifications into a single summarizing notification, regardless of your notification method (phone, SMS, email, or push). All incidents that are in the triggered state (i.e. not acknowledged or resolved) will be included in the bundled notification.

Here are examples of how each contact method will notify you, in the event of near-simultaneous incidents. For this example, assume I've triggered 5 incidents on a service named "Nagios" within a 1-minute time frame. 

Phone

You have 5 triggered incidents on Nagios. Press 4 to acknowledge all incidents. Press 6 to resolve all incidents. Press 0 for help or press * to repeat this message.

SMS

ALRT: #100,#101,#102,#103,#104 on Nagios. Reply 14:Ack all, 16:Resolv all

Email

You are assigned 5 triggered incidents in PagerDuty:
Please visit the following URL to manage these incidents.
https://fake-name.pagerduty.com/dashboard

1) Incident #100
   Opened on: Jan 1 at 12:00pm PST
   Service: Nagios
   Description: Emergency in the server room
   Link: https://fake-service.pagerduty.com/i/100

 

Another way to minimize the number of notifications that you receive is to adjust your notification rules so that you are notified of an incident several minutes after it is assigned to you as opposed to immediately after it is assigned to you. Adjusting the notification rules will allow more time for incidents to aggregate and you will receive fewer notifications during a notification storm.

Have more questions? Submit a request

Comments

  • Avatar
    Vadym Mashkov

    Good article, thanks!

    Is is option enabled by default?

  • Avatar
    Jonathan Curry

    Hi Vadym,

    Yes, notification bundling is enabled for all users on PagerDuty; there's nothing extra you need to do to take advantage of this feature.

  • Avatar
    Vadym Mashkov

    Cool. thanks for answer!

    Can we adjust amount of messages for it? As for me 100 notifications per minute is a very big value :)

  • Avatar
    Jonathan Curry

    There's no configuration options for bundling at this time, but the threshold isn't 100 notifications either - that's just an example of an extreme situation. Notification bundling will happen with a few as 2 notifications coming in at the same time.

    I hope this clears things up!