Email integrations allow users to integrate PagerDuty with any service capable of sending email alerts. PagerDuty provides a number of features to assist with administering email integrations, which can be broken down into the two following broad categories:

• Email Filters
• Email Management

Email Filters

Email filters allow you to determine which messages your email integration should accept. Only messages that meet your specified criteria will trigger an incident. Messages that do not pass the filter will be discarded.

Configure an Email Filter

To set up an email filter, in the web app:

1. Go to Services Service Directory and select the service with the desired email integration.
2. Select the Integrations tab next to the email integration you want to edit click Edit Integration.
3. In the section Email Filters, make a selection from the dropdown, depending on your needs:
   • Accept all incoming email: PagerDuty will not filter out any emails that are sent to your integration email address.
   • Accept email only if it matches ONE OR MORE rules below: Emails will only be accepted if they match at least one filter rule.
   • Accept email only if it matches ALL of the rules below: Emails will only be accepted if they match all filter rules.
4. Configure appropriate rules:
   • The dropdowns for subject, body, and from default to is anything. Next to your desired field, select the appropriate action (matches the regex or does not match the regex) and enter a regular expression.
5. Optional: If required, click Add New Rule and repeat the previous step to add more filtering criteria.
6. Click Save.

Delete an Email Filter

1. Go to Services Service Directory and select the service with the desired email integration.
2. Select the Integrations tab, then next to the email integration you want to edit, then click Edit Integration.
3. Click in the top right of the email filter you’d like to delete.

4. Click Save changes.

Email Filter Example

Below is an example of how to configure email filters based on an event’s critical/noncritical status. In this example, you have a system that sends emails with the subjects CRITICAL and NONCRITICAL to PagerDuty, but you only want those with CRITICAL in the subject line to trigger incidents.

1. In the web app, go to Services Service Directory and select the name of the service with an email integration.
2. Select the Integrations tab click to the right of the email integration you want to edit click Edit Integration.
3. Under Email Filters, select Accept email only if it matches ALL of the rules below.
4. Add two filter rules:
   1. For the first rule, enter The email subject matches the regex CRITICAL.
   2. For the second rule, enter The email subject does not match the regex NONCRITICAL.

• The option Accept email only if it matches ALL of the rules below ensures that both rules will be applied to incoming events. A second filter for NONCRITICAL is needed, since it contains the substring CRITICAL. In other words, the second rule prevents messages with the subject NONCRITICAL from triggering incidents, since it also matches the regex CRITICAL. Please also note that regex is case sensitive.

5. Click Save changes.

Email Management

By default, PagerDuty opens a new incident every time an email is received at the integration address of an email integration. Depending on how your monitoring tool works and your organization’s needs, this behavior might not be desirable.

As a first step, we recommend configuring one or more email filters to ensure that your service only accepts valid email events.

When your email filters determine that an event is valid, you can use email management rules to fine tune how the events behave. For example, the email management feature allows you to create custom rules that parse inbound messages and automatically resolve incidents.

• Configure When Incidents or Alerts Trigger
• Trigger and Resolve Alerts/Incidents

Configure When Incidents or Alerts Trigger

You can control when PagerDuty creates new incidents or alerts in response to emails by adjusting your email integration’s settings in the web app:

1. Navigate to Services Service Directory.
2. Select the name of the service with the desired email integration Integrations tab to the right email integration click Edit Integration.
3. In the section Email Management, select your desired setting:

• Open a new incident/alert for each new trigger email subject (default): Emails with the same subject line will deduplicate. For example, if PagerDuty triggers an incident in response to an email with the subject “Host Down”, subsequent emails with the same subject will not trigger a new incident. Rather, the new email will be appended to the existing incident’s timeline.
• Open a new incident/alert for each trigger email: Each email sent to the integration email address opens a new incident.
• Open a new incident/alert only if an open incident does not already exist: The integration will only trigger one open incident at any time. If an email is received while the integration already has an open incident (regardless of its subject line), it is appended to the existing incident’s activity log.
• Open and resolve incidents/alerts based on custom rules: This option is explored in-depth in the section Trigger and Resolve Alerts/Incidents.

4. Click Save.

Trigger and Resolve Alerts/Incidents

Selecting the option Open and resolve alerts/incidents based on custom rules allows you to configure rules that determine when an incident should trigger and resolve, based on the message’s content.

There are two components to configure, and both are accessed in the web app by navigating to Services Service Directory select the name of your desired service Integrations tab click next to the desired email integration click Edit Integration:

• Create a Trigger Rule
• Create a Resolve Rule

Create a Trigger Rule

1. From the dropdown, select trigger.
2. Select Any (default) or All depending on whether you’d like to trigger an incident when any or all following conditions match.

3. Determine the matching condition:
   • Select from the dropdown whether you’d like to match on the email from address, subject or body.
   • Select from the following matching options:
     • contains
     • does not contain
     • matches the regular expression
     • does not match the regular expression
     • exactly matches
     • is anything except
   • Enter your desired matching criteria
4. Optional: If you’d like to configure more than one trigger condition, click New Condition and repeat step 3.
5. Configure an incident/alert key: It is important that this information matches in the corresponding resolve rule.
   • In the section Alert/Incident Key, select subject or body from the dropdown.
   • Select from the following matching options:
     • all text between
     • this regular expression
     • everything
     • all text after
     • all text before

📘

Use Regex to Extract an Incident/Alert Key

If you plan to use a regular expression to generate the unique key for an incident/alert, please refer to the section Use Regex to Extract an Incident/Alert Key for more information.

6. Optional: Configure Custom Fields. This feature can be helpful if your trigger emails contain information that you’d like to be included in incident details.
   • Enter a Name for the field.
   • Configure the Value:
     • Select subject or body from the dropdown.
     • Select from the following matching options:
       • all text between
       • this regular expression
       • everything
       • all text after
       • all text before
     • You may add more custom fields by clicking New custom field and repeating the steps above.
7. Click Save changes.

Create a Resolve Rule

A resolve rule is required if you’d like incidents to automatically resolve when PagerDuty receives a valid email event.

📘

Auto-Resolve Using an Alert/Incident Key

For a pair of email management trigger and resolve rules to work together, the alert/incident key in both rules must be a perfect match. The alert/incident key is what allows your service to pair events and apply resolve rules to open incidents.

1. Select Add Another Rule at the bottom of the page.

2. Configure the resolve rule similarly to the trigger rule, however you will select resolve from the dropdown, and configure the condition(s) that should resolve incidents/alerts. The information for Incident/Alert Key should exactly match the corresponding trigger rule.

📘

No Matching Open Incident

If an email matches your "resolve" rule but there is no open incident for that email to resolve OR the email's incident key does not match an existing open incident, then this rule will not be applied.

Create Multiple Rules

In some cases, you may want to create more than one trigger and/or resolve rules. To do this, click Add Another Rule.

When you create multiple email management rules, PagerDuty will attempt to apply the first rule to the incoming email. If it does not match it will attempt to apply the second rule, and so on. The hierarchy continues in this order until it reaches your last rule. There is a limit of ten rules per service.

You can change the order rules are applied in by clicking Move up and Move down buttons on each rule.

Configure Default Behavior If No Rules Match

If an email does not match any email management rules, you can configure whether you’d like to create a generic incident or discard the email.

Select your preference from the dropdown:

• create a generic incident
  • Generic incidents will use the email’s subject line as the name of the incident.
• discard it

Delete an Email Management Rule

1. Go to Services Service Directory and select the service with the desired email integration.
2. Select the Integrations tab, then next to the email integration you want to edit, then click Edit Integration.
3. Click in the top right of the email management rule you’d like to delete.

4. Click Save changes.

Email Management Example

In this example, you want to trigger an incident when PagerDuty receives an email (from a monitoring system, for example) that contains DOWN in the subject line. The body of email is always formatted such that a numeric string representing the host follows “Host: “, which can be used as the incident/alert key. We can use a [regular expression] to for this: Host: (\d+).

1. Create a trigger rule that captures the scenario outlined above.

2. Create a resolve rule

With the email management rules in place, when ​​PagerDuty receives an appropriate email that meets your trigger rule’s conditions, an incident will trigger.

Similarly, when the email integration receives an appropriate resolve message, the incident will automatically resolve

Troubleshoot Email Management Rules

Here are some solutions to issues that may come up when working with email management rules:

• Replies and Email Forwards Trigger a New Incident
• Trigger Emails are not Accepted
• Resolve Emails are not Resolving Incidents
• Capture Group Error Message
• A Regular Expression is Not Matching the Email Body as Expected

Replies and Email Forwards Trigger a New Incident

By default, PagerDuty email integrations accept any messages from any sender, including messages addressed to multiple recipients. When someone replies to all recipients on an email with recipients that include the email integration address, an email will be sent to the email integration address. This can result in a new incident triggering, in addition to the original incident.

You can configure an email management trigger rule to prevent replies and forwards from triggering duplicate incidents:

1. Configure the rule to trigger an alert/incident if Any of the following conditions apply.
2. Configure a condition: The email subject matches the regular expression: .*
3. For the incident/alert key, have the rule state: In the email subject, match this regular expression: (?:(?i)re: |fwd: )?(.*).
   • This regular expression is case insensitive, so any variation of "RE:" or "FWD:" will deduplicate into the open incident.
4. Click Save changes.

📘

Case Insensitive

This regular expression is case insensitive, so any version of "RE:" or "FWD:" will deduplicate the email trigger into the open incident.

Replies and forwards will now be appended to the existing incident’s timeline.

Trigger Emails are not Accepted

First, check your email filters. PagerDuty evaluates email filters before management rules in order to reduce noise and ensure all emails come from a verified source.

If you think the filters might be dropping email, try setting them to Accept all incoming email and test again. Keep in mind that changes to your email filters will not affect events that were received before the changes were made.

If you are using email filters, double-check your regular expressions to ensure that desired emails are accepted.

If emails are still not triggering incidents as you’d expect, check the integration’s default behavior; Create a generic incident should be selected. This is the default setting, but if another user changed it, for example, any unmatched emails will be discarded.

Resolve Emails are not Resolving Incidents

Three things are required to successfully resolve an incident:

1. A rule that triggers an incident in PagerDuty.
2. A rule that resolves an incident in PagerDuty.
3. A shared incident key that can be extracted from both emails.

Verify that your rules are correct. PagerDuty will display the extracted incident key on an incident’s details page. Use this piece of information to verify that the key is the same in both the trigger and resolve emails.

Specific reasons a resolve email might not be successfully resolving:

• There is an issue with a regular expression extracting the incident/alert key.
• There is an issue with the email body’s encoding (for example, if it was UTF-8 encoded, but contained the byte \xFF).
• There is a trailing space, for example, in either your trigger rule or resolve rule, but not in both. This means that one email has an incident/alert key with an additional space, and therefore will not match.

Capture Group Error Message

If you receive the following error, it means that you need to specify a capture group in your regular expression:

Email handler email action rules value extractors matcher must have at least one capture group

Oftentimes, this can be resolved by enclosing your regular expression in parentheses.

A Regular Expression is Not Matching the Email Body as Expected

If a regular expression that should match the email body is not working for some reason, check to see whether emails are sent in both plain text and HTML, or plain text. If the emails are only sent in HTML, the HTML tags may be interfering with the regex’s pattern matching.

If the email also includes a plain text version, then the rule will still be able to find a match. However, if the emails are being sent as HTML-only, note that the regular expression may not match because there are HTML tags and elements interspersed within the text content that makes up the searched-for pattern. It may help to view the raw email that was checked for a match in the email rules.

📘

View the Email Source Code

Go to View Message in the incident detail view. This will take you to a page displaying the full details of the original event sent in. The URL should be formatted as follows:

https://{subdomain}.pagerduty.com/incidents/{incident_id}/log_entries/{log_entry_id}

Next, click View Raw Message to see the email’s source code, including headers.

While viewing the source code, note that email management rules that operate on the body of the email use the raw body as input, after decoding (i.e., if the body’s Content-Transfer-Encoding is base64, it will be decoded from Base64, or if it's quoted-printable, then QP escape sequences will be replaced with the actual characters that represent the message’s underlying content). With this in mind, to view the true source of the email that the regular expressions are compared against, one must decode the text.

Base64-encoded Text

Base64-encoded messages will show the following information in their header:

Content-Transfer-Encoding: base64

Base64-encoded messages are not humanly readable, but are easily decoded into plain text using readily available software. We recommend the base64 command line program, which is found on many Linux-based and Unix-based systems, including macOS.

📘

Example of Base64-encoded text

Decoded (Regex runs on this text):

All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy.
The quick fox jumps over the lazy brown dog.

Encoded (Regex does not run on this text):

QWxsIHdvcmsgYW5kIG5vIHBsYXkgbWFrZXMgSmFjayBhIGR1bGwgYm95LiBBbGwgd29yayBhbmQg
bm8gcGxheSBtYWtlcyBKYWNrIGEgZHVsbCBib3kuClRoZSBxdWljayBmb3gganVtcHMgb3ZlciB0
aGUgbGF6eSBicm93biBkb2cuCg=

Decoding with base64:

base64 -D < file-containing-base64-message.txt
# or this:
cat file-containing-base64-message.txt | base64 -D

Quoted-Printable Text

Quoted-Printable messages will show the following information in their header:

Content-Transfer-Encoding: quoted-printable

Messages with Quoted-Printable encoding can be easier for humans to read, but a variety of tools are available for decoding them.

When reading raw Quoted-Printable encoded text, please be aware of escape sequences that constitute part of the encoding, or that are a representation of a character and not the literal character. Non-ASCII characters will be represented as escape sequences, as will equals signs (=).

Furthermore, lines are limited to 76 characters. If a line’s length needs to be limited, a line may be encoded as two or more lines with a line break and an equals sign at the end of every line except the last in the series.

📘

Example of Quoted-Printable encoding

Decoded (Regex runs on this text):

All work and no play makes Jack a dull boy. All work and no play makes Jack a dull boy.
The quick fox jumps over the lazy brown dog.
E = mc^2

Encoded (Regex does not run on this text):

All work and no play makes Jack a dull boy. All work and no play makes Jack=
 a dull boy.
The quick fox jumps over the lazy brown dog.
E =3D mc^2

Regular Expressions

Regular expressions (regex) are a standard, well-documented way to match text. In the context of PagerDuty email integrations, you can use regex to finetune your email filters and email management rules.

Use Regex to Extract an Incident/Alert Key

If you use a regular expression to extract the incident/alert key, you will need to use a capture group. In regex, a capture group is defined as anything between a set of parentheses, (... ).

Capture Group

A capture group will tell PagerDuty to create an incident/alert key from the text between a set of parentheses, (...). If the unique identifier within the capture group is likely to change (e.g., a numeric string like a ticket ID or host ID), you can use \d+, which tells PagerDuty to capture all subsequent digits.

Similarly, if you want to capture a specific number of digits you could use \d to stand in place of each digit. For example, you could use \d\d\d\d or \d{4} if the unique identifier always contains 4 digits.

Capture Parentheses in the Unique Identifier

In some instances your unique identifier may contain parentheses. For example, Zendesk ticket IDs typically contain parentheses: This ticket (#27415) has been assigned to group ‘Support’, of which you are a member.

To capture the parentheses in a regex, use a backslash (\) before each parenthesis, for example: \(#\d+\). The backslash acts as an escape character that tells PagerDuty "the following character should be treated as text."

HTML Emails and Content Encoding

Email management regular expressions are compared on the text of the email after decoding, per the encoding specified by Content-Transfer-Encoding. However, Content-Type is ignored. In other words, if, after decoding a base64-encoded email, the underlying message is HTML, and there is no plain text copy of the email included, then the regular expression will run on the HTML code, not on the text that would result after rendering the HTML or stripping out HTML tags.

This means that if emails are sent only as HTML (Content-Type: text/html), regular expressions written to match the text content alone may not work. HTML tags, hidden line breaks, and whitespace can potentially interfere with your regular expressions’ ability to match the input.

See Troubleshooting for further details.

Case Sensitivity

Regular expressions are case sensitive, so "DOWN", "down", and "Down" are all considered different strings and will not all match against the same regex. You can make your email management rules case insensitive by adding (?i) to the beginning of the line, for example: (?i)(critical)

This will capture all cases: "critical", "CRITICAL", "cRiTicAL", etc.

📘

Tip

The case insensitivity modifier (?i) works in email management rules, however it is not supported with email filters. With this in mind, we suggest using a pipe character (|) to capture different upper/lowercase strings for email filters, for example: (Down|DOWN|down)

Test Your Regex

We recommend Regex101 to compose, edit and test your regular expressions.

To test a regular expression in Regex101, enter your test string (in this example, you’d like to extract the node, A74U-138, from the email body and use it as the incident/alert key) and make sure that the regex show a Group 1 match.

👍

Regex Testing Tip

When using the external site Regex101 to test your regex, please select the following options:

• Golang flavor
• multi line
• single line

If your regular expression matches on the sample text you enter into Regex101, it should work for the same text in PagerDuty.

Specifics of PagerDuty's Capture Group Implementation

PagerDuty's email management rules use Google's RE2 for regular expression handling, and additionally adds some custom behavior. Some specific things to be aware of:

• Nested capture groups are not supported. The following regular expression will generate an error when saving the service: (([0-9])[0-9]).
• If you define multiple capture groups, we will concatenate what is captured in each group together, using a - character. For example, if you match ([a-z])([a-z]) against ab, PagerDuty will extract this as a-b.
• Two options are always added to your regular expression (documented in the RE2 documentation) when extracting data:
  • s, which means that the . special character will also match newlines (\n).
  • m, this is called multi-line mode, which causes ^ and $ to match the beginning and end of lines, in addition to beginning and end of the entire text.
  • A consequence of these two behaviors and regex greediness is that (.*)$ will match everything until the end of the document. We recommend using ([^\n]*) if you’d like to match everything until the end of the line.

Regex Examples

Here is an example of a common use case using a regular expression:

Your PagerDuty service receives emails from a monitoring tool, however you only want to trigger incidents if the subject line starts with “CRITICAL” or “SEVERE”. The regular expression you would use to match incidents of this type would be: ^(CRITICAL|SEVERE).

The ^ character means the subject line starts with this, and (CRITICAL|SEVERE) means the starting word can be either "CRITICAL" or "SEVERE".

Other Common Regex Examples