Trigger and resolve alerts with email management rules

Follow

By default, PagerDuty opens a new incident every time an email is received at the integration address of an email integration. Depending on how your monitoring tool works, this behavior might not be desirable.

The first line of defense in limiting unnecessary noise in email integrations are email filters. These filters select which of your inbound events will become events on your service, and which events will be discarded. 

Once an event is accepted by your email filters, email management rules may be used to determine the way accepted events will behave. PagerDuty's email management feature allows you to automatically resolve incidents from email integrations by creating custom rules to parse messages and specify how they should be handled.

 

Control when a new incident is triggered

You can control when PagerDuty creates new incidents in response to emails by adjusting your integration settings.

To set how new incidents will be created in your email integration:

  1. Go to the Configuration menu and select Services.

  2. Click on the name of the service that contains the integration, then go to the Integrations tab and click the name of the integration to find the Edit button

  3. Under the Email Management section, choose a new setting, then click Save.

The following settings are available:

  • Open a new incident for each trigger email: Each email sent to the integration email address opens a new incident.

  • Open a new incident for each new trigger email subject: Emails with the same subject line de-duplicate. For example, if PagerDuty previously opened a new incident because it received an email with the subject “Host Down”, a second email with the same subject will not cause create a new incident. Instead, the new email will be appended to the existing incident’s activity log.

  • Open a new incident only if an open incident does not already exist: The integration can have only one open incident at any time. If an email is received while the integration already has an open incident, it is appended to the existing incident’s activity log.

  • Create and resolve incidents based on custom rules: This option will be explored in-depth in the next section of this article.

The last two options above will append incoming emails matching the criteria to existing incidents, as shown below:

 

Alert keys and conditions

Selecting the last option -- create and resolve incidents based on custom rules -- in your email management fields allows you to automatically resolve incidents from email integrations by creating custom rules to parse messages and specify how they should be handled.

Email management uses a set of rules that is applied to any incoming email. Each rule has two components:

  1. condition that tells PagerDuty if the email should create an incident, or resolve an existing incident

  2. An alert key that tells PagerDuty how to match one email with another email.

In the example above, the incident key has been highlighted in green. The conditions are in caps: DOWN for trigger, UP for resolve.

 

Creating your rules

  1. Specify if PagerDuty should trigger or resolve an incident and the criteria of the email

  2. Specify the alert key - the alerts key de-duplicates emails that have the same identifiers and will append the email to an existing incident

  3. Optional - you can add additional criteria in the Custom Field if you'd like to extract additional data from the email. This can be useful with certain integrations, like ZenDesk.

Note: If an email matches your "resolve" rule and there is no open incident for that email to resolve OR the email's incident key does not match an existing open incident, then we will discard the email (as opposed to having the email trigger a new incident).

Just as with email filters, you can use regular expressions to create complex management rules - however regular expression are not mandatory. You can review how to create a regular expression in Regular Expression Tips & Examples.

In some cases you may want to create an additional rule by clicking Add Another Rule. When you create multiple rules, PagerDuty will apply the first rule to the incoming email, and if it doesn't match it will attempt to apply the second rule. The hierarchy continues in this order until it reaches your last rule. There is a limit of ten rules per service. Rules may be re-arranged by clicking the Move up and Move down buttons on each rule.

Finally, if an email doesn't match any of the rules, PagerDuty will either create a generic incident or discard the email. Use the drop down menu to specify your preference - you can either create a generic incident or discard it (the email).

Have more questions? Submit a request

Comments