The following are the most common issues, and their troubleshooting steps, for the ServiceNow v7 integration.

PagerDuty App Does not Appear in the System Menu

If you’re having trouble finding the PagerDuty app in ServiceNow’s system menu, please check the following first:

• If you just installed the app, try a hard refresh in your browser, or log out and log in again, as the left menu may need to reload after installation.
• Make sure the PagerDuty app has been added to your current instance from the ServiceNow store.
• Install the app to your current instance by going to System Applications Applications Downloads tab.

Conflicting Plugins

If you’ve completed the steps above and are still unable to see the PagerDuty app, check to see if you have any plugins that could prevent the installation from succeeding. For example, if you have installed a plugin that encrypts attachments in ServiceNow, we’ve seen cases where it looks like the installation completes when it has not actually done so. If you identify such a plugin, please try deactivating it and attempting installation again.

ServiceNow Incidents and Updates do not Appear in PagerDuty

1. In ServiceNow, check the REST API connection by navigating to PagerDuty Actions Test REST API Connection. Check that trigger conditions are met.
2. Check your primary ServiceNow app settings under PagerDuty Configuration PagerDuty Settings. If auto-provisioning is disabled, check the value in the field Default PagerDuty User ID to use if auto-provisioning is disabled.
3. See the section When is an incident sent to PagerDuty?, in our ServiceNow v7 FAQ page.

🚧

Default PagerDuty User ID

The Default PagerDuty User ID configuration property in the ServiceNow app needs to be a PagerDuty ID, rather than an email address or ServiceNow user ID. PagerDuty IDs are uppercase alphanumeric strings that begin with P (see: Where do PagerDuty field values come from?).

There should be only one user that has the given PagerDuty user ID in your ServiceNow instance, and the user’s email address should match the corresponding PagerDuty user with that ID. If they do not match, some API calls may not succeed.

An email address is required for the REST API calls that create PagerDuty incidents. With that in mind, if this information is incorrect, the requests may not succeed.

Incidents in ServiceNow are not Updated or Created from PagerDuty

Most data and event synchronization from PagerDuty to ServiceNow happens via webhooks sent from PagerDuty to ServiceNow. For this reason, if you notice incidents are not updating or getting created from PagerDuty, a good place to begin troubleshooting is to review the webhook import process.

Check webhook authentication and permissions

To test all settings, go to PagerDuty Actions Test ServiceNow User Authentication. You should see the message:

x_pd_integration (PagerDutyTest): ServiceNow user authentication test successful (200)

If you see the following:

x_pd_integration (PagerDutyTest): ServiceNow user authentication test failed (403)

1. Ensure that the ServiceNow user for authentication and ServiceNow password for authentication configured under PagerDuty Configuration PagerDuty Settings are valid credentials for a ServiceNow web service access user.
2. Ensure that the password is consistent between the user and the setting in the PagerDuty Configuration properties.
3. Ensure that the user for webhook authentication has the following roles:
   • itil
   • rest_service
   • x_pd_integration.admin

If the password or username have changed in ServiceNow, you must also update the credentials in the ServiceNow extensions on your PagerDuty services. Alternately, in all versions 3.5 and later, you can delete the extensions, set the value of the PagerDuty Webhook field to blank in your provisioned assignment groups and configuration items, and then provision them again.

Check Extensions on PagerDuty Services

• Check that the extension’s URL is correct. The host it points to should match the same ServiceNow subdomain that you integrated PagerDuty with.

• If you have provisioned your services and webhooks in PagerDuty from a development or staging instance of ServiceNow, you will need to update your extension URLs to reflect your production ServiceNow instance. If you do not take this step, the bidirectional integration will not work because webhooks would be sent to the development/staging ServiceNow instance.

Read more about PagerDuty service extensions, please read Extensions and Add-ons.

Check That Assignment Groups Match the Escalation Policy

• If a ServiceNow incident isn’t updating after changes happen on the corresponding PagerDuty incident, make sure the relevant PagerDuty escalation policy has a corresponding assignment group in ServiceNow.

• Ensure that the appropriate assignment group has its PagerDuty Escalation field set to the escalation policy’s ID (for example, PXXXXXX).

• You may need to add the two PagerDuty fields to the group form or list view to review their values.

Check Webhook Import Records

Check to see if the webhook messages have been saved in ServiceNow. In ServiceNow, go to PagerDuty Import Rows Webhook Import Rows and look for records with a creation time that coincides with when the incident in PagerDuty was triggered, acknowledged or resolved.

Check for Required Fields

If you can see that webhooks are getting through from PagerDuty to ServiceNow, but are not creating or updating an incident, try manually closing an incident in ServiceNow. If manually closing an incident requires you to enter values for certain fields, your required fields have likely been customized. Create inbound field rules so that required fields have a default value.

Check for Interfering State Field Values

Check to see if you have any custom choices set up for the State field in the ServiceNow incidents model:

1. Right-click on the State field’s label (e.g., New or Resolved).
2. Select Configure Dictionary.
3. Switch to Advanced view.
4. Select the Choices tab and look for entries with higher numbers in the Value field.

The integration follows the same convention as ServiceNow’s out-of-box state values. Higher numbers imply a more advanced state (i.e., 6 or higher represents done, resolved, or some variant). If the state of the incident that did not automatically resolve is a higher numeric value, but the incident did not resolve, it is because it was treated as already resolved.

To correct this, you may need to modify custom choice values for the State field to be lower than 6, so that they are treated as an unresolved state.

Check For Interfering Data Policies

Data policies, which allow the ServiceNow administrator to create validation rules on tables, can stop the webhook import process from committing PagerDuty changes to the ServiceNow incident record, and thus prevent the integration from automatically resolving incidents.

When a data policy does this, there will not be any clear indication of it in the application-scoped logs. There will, however, be an error-level log entry in the system logs created at the time of the webhook import, with a message beginning that begins Data Policy Exception.

As of version 4 of the integration, inbound field rules give you the ability to automatically fill fields in the incident, and this can be used to fill fields as needed in order to satisfy data policies.

To address this:

1. Navigate to System Policy Rules Data Policies.
2. In the list view, create a new filter on the Table field: =incident.
3. Make note of all the fields and constraints.
4. Go to PagerDuty Configuration PagerDuty Inbound Field Rules.
5. For each incident field that was part of the data policy’s conditions:
   • Create a new inbound field rule.
   • Set the webhook type to Acknowledge or Resolve, based on which action is experiencing an issue.
   • Enter a default value for the field, or dynamically based on the webhook payload.

For more information, please see Advanced ServiceNow v7 Configuration: Inbound Field Rules.

Inbound Field Rules not Filling Target Fields with Desired Values

The most common causes include:

• A business rule or some other process is overwriting the changes to the field after it is initially set by the field rule.
• The inbound field rule only applies to a specific webhook event type (e.g., trigger/resolve), and the change didn't occur at the specified time because the relevant webhook type came through at a different time.
• The inbound field rule is disabled.

To troubleshoot:

1. In ServiceNow, go to PagerDuty Configuration PagerDuty Inbound Field Rules and verify the following:

• For the inbound field rule in question, Enabled is true.
• The PagerDuty webhook type is the expected type.

2. Go to System Definition ​​ Business Rules and look for all rules that modify the field as follows:

• Enable the field Set field values in the list view.
• Search filter for table: =incident.
• Filter out all rows where Set field values is blank.
• If there is more than one page of results, search for all records where Set field values contains the column name, e.g., *business_service. If any results modify this field, it may explain why the field is not getting set according to the inbound field rule; check that the business rule is not overwriting the field.
• If nothing turns up, look at all business rules that run a script and check whether any scripts set the value of the field in question.

Some User Actions in PagerDuty are not Updated in ServiceNow

A common cause of this type of issue is that the email address for the PagerDuty user and corresponding ServiceNow user do not match. Check the user's email address in ServiceNow to make sure it matches the user's email address in PagerDuty. Also confirm that the user is provisioned in both ServiceNow and PagerDuty.

Incident Notes do not Sync

• The user posting the note might not be provisioned in PagerDuty. In other words, it is possible that the user is not assigned a PagerDuty user ID in ServiceNow. This issue can also occur if automatic provisioning is disabled, and one of the following applies:
  • Under Configuration PagerDuty Settings, there is no value set for the option Default PagerDuty User ID to use if auto-provisioning is disabled.
  • There is an incorrect or invalid value set for this option. The value must be a PagerDuty user ID, which is an uppercase alphanumeric code at the end of a given user's profile URL (for example, PXXXXXX).

Webhook Migration

If you do not see the option to migrate webhooks, you can re-enable the menu item and run the migration. To do this:

1. In ServiceNow, go to System Definition Application Menus.
2. Search and select “PagerDuty” to go to the application’s edit screen.
3. Under the menu items, find MIGRATE WEBHOOKS and check its box.
4. In the Actions on selected rows dropdown, select Change active state.
5. Refresh the page and go to the PagerDuty application in the system menu.

The menu item should now be visible.

ServiceNow Sends Too Many Emails or Worknote Updates on an Incident

We’ve seen cases where ServiceNow sends hundreds, or even thousands of emails or worknote updates on an incident. This is usually due to a feedback loop, which can result from custom business rules or automation in ServiceNow. It is also possible to use webhooks to inadvertently create a feedback loop between ServiceNow and PagerDuty.

This can happen if there are any business rules or other automated processes that alter the assignment group of the ServiceNow incident, and caution is not exercised when manually setting the integration fields. To give a high-level overview of an example of how this can possibly happen, let us assume the following configuration:

• There is a business rule that automatically changes the assignment group to B if it is placed into assignment group A.
• Both assignment group A and B are tied to PagerDuty escalation policy Z.

A feedback loop occurs in the following scenario:

1. Incident changes assignment group to, or is created in, B.
2. A PagerDuty incident triggers and is assigned to escalation policy Z.
3. A webhook fires and escalation restarts (webhook type incident.delegate or incident.trigger).
4. Webhook import in ServiceNow assigns incident to assignment group A.
5. Automation changes assignment to B.
6. The loop repeats.

If your configuration matches or is similar to the scenario described above, you’ll need to make adjustments so that the loop is broken.

Duplicate Incidents

ServiceNow Creates Duplicate Incidents in PagerDuty

This can happen when a workflow circumvents the methods designed to avoid triggering duplicate PagerDuty incidents.

📘

How the Integration Works

The integration uses the PagerDuty Incident field in the ServiceNow incident as a marker to prevent race conditions. Initially it is populated with a placeholder value ---waiting--- and is later filled with the PagerDuty incident ID once it is available.

The placeholder value is filled in immediately when the business rule runs, and the business rule's conditions exclude the case where the field has a value that starts with ---. With this in mind, if two updates happen in succession, before the incident triggers in PagerDuty, the business rule won't run twice.

The incident is updated again when the PagerDuty REST API is called to trigger the incident. To prevent other updates from triggering the same business rule more than once, current.update() is called to commit the incident with the placeholder value to the database.

Assuming that there have been no customizations to the out-of-box PagerDuty integration, the most likely cause is an automation, i.e., a separate business rule, that is triggering a second automatic update on the incident record after the initial one. The business rule PD Trigger PagerDuty Incident runs with order = 100, meaning that it runs after changes are committed to the database. If two updates happen asynchronously in rapid succession, they risk triggering the business rule that creates a PagerDuty incident twice.

To resolve this, first you will need to find out how the secondary updates take place:

1. In ServiceNow, go to System Logs Events.
2. Find events named x_pd_integration.trigger_incident around the time duplicate incidents triggered.
3. Note the values of:
   • param1, the last worknotes entry.
   • param2, a comma-delimited list of extra parameters: current user sys_id, the operation (insert or update), and a boolean for whether the value of the assignment group or Configuration Item changed.
   • User ID
   • User name
4. Identify the event that was raised by the automated process based on how the event was submitted, vis-à-vis the available attributes.
5. Find some attribute of the automated update operation that can be used to distinguish it from others, e.g., if it was made by a human.
6. Add conditions to the business rule PD Trigger PagerDuty Incident that exclude updates which have the distinguishing characteristics of the automated updates.

PagerDuty Creates Duplicate Incidents in ServiceNow

📘

How the Integration Works

The PagerDuty App receives webhooks and saves them in an import table, and uses an import transform map to translate them to updates to existing incidents or insertion of new incidents. It uses the PagerDuty Incident field in the ServiceNow incident to identify existing incidents

Coalesce Disabled

You may also see this behavior if the import transform map is not set to coalesce on the incident ID. Verify that coalescing on the incident ID is enabled in the import transform map:

1. In ServiceNow, go to PagerDuty Import Rows Webhook Import Service.
2. At the bottom of the page, select PagerDuty Webhook Import under Web Service Transform Maps.
3. At the bottom of this page, find the section Field Maps.
4. In the mapping for id x_pd_integration_incident, ensure that Coalesce is set to true.