PagerDuty Agent is an open-source utility for integrating standalone/on-premise monitoring tools with PagerDuty. This guide offers help for troubleshooting the most common issues.

• Basic Troubleshooting
• Outqueue Troubleshooting
• Network Troubleshooting
• Testing TLS
• Proxy Troubleshooting

Basic Troubleshooting

If you have trouble sending events to PagerDuty using the agent, run the following checks first:

• Verify that the pdagent service is running.. If the service is not running, you can start it with the command systemctl start pdagent (on systemd-based systems) or sudo service pdagent start (on SysV init-based systems). Any enqueued events that accumulated while the service was stopped will be sent when the service starts again.
• Check the service integration key: Make sure that the key is for the correct service and that it has been entered correctly.
• Send a test event using pd-send. Remember to replace the key used in the example with your own PagerDuty service integration key.
• Check the logs for any errors, including events that couldn't be submitted to the Events API. These are typically found under /var/log/pdagent/pdagentd.log.

Installation Error: pdagent.service not found

Sometimes on a fresh install of the PagerDuty Agent, you might see the error message Failed to start pdagent.service: Unit pdagent.service not found. This is due to an error during installation where the file pdagent.service was not copied to the proper directory.

In order to fix this, you can copy the pdagent.service file into the proper directory with the following steps:

1. Find the file pdagent.service with the command:

sudo find / -name pdagent.service

2. Copy the location of the file.
3. Copy the file to the correct directory the command:

sudo cp \<PATH_TO_FILE> /etc/systemd/system

4. Restart the agent with systemctl start pdagent or service pdagent start, respectively, depending on whether your operating system uses systemctl or service.

Outqueue Troubleshooting

In addition to using pd-queue to manage the queue, the agent's outqueue directories often contain useful troubleshooting information. You can also manually clear the queue by working directly with the files.

Each file in the outqueue directories contains the body of an HTTP POST request that will be made to the PagerDuty Events API. You can examine these files to see what your monitoring tools are submitting to the agent. File names follow the naming convention {UNIX timestamp}_{integration key}.txt.

Outqueue Directories

The queue directories, located in /var/lib/pdagent/outqueue, include:

🚧

Event Expiration and Queue Cleaning

Events are deleted from the outqueue directories at a regular interval once their age exceeds a predefined length, in seconds. The default age threshold is one week, and the default cleaning interval is three hours. Both the interval and the age threshold are defined in the PagerDuty Agent configuration file, which should be located at /etc/pdagent.conf:

• cleanup_interval_secs: The interval between when the queue is cleaned.
• cleanup_threshold_secs: The age, in seconds, after which events will be deleted.

Manually Purge the Queue

In the event of a long network outage, where a large number of events accumulate in the agent’s outqueue, you can manually clear out the queue to avoid a heavy cluster of PagerDuty incidents when network connectivity is restored.

To clear the queue, run the following commands:

sudo rm -vf /var/lib/pdagent/outqueue/pdq/*
sudo rm -vf /var/lib/pdagent/outqueue/tmp/*

Network Troubleshooting

The host that is running PagerDuty Agent should be able to make an HTTPS connection to events.pagerduty.com. With this in mind, if the LAN does not have DNS service, there is no route to the internet, or the local ACL (if any) does not permit connecting to remote hosts, the agent will not be able to send data to PagerDuty.

The following items can help resolve network issues:

• DNS
• Reachability
• TCP Connection

DNS

In the host’s command line, run the following command:

dig events.pagerduty.com

If you see a status response SRVFAIL or NXDOMAIN, the issue is most likely with DNS.

Reachability

Our Events API hosts should respond to Internet Control Message Protocol (ICMP). Run the following command:

ping -q -c 3 events.pagerduty.com

If you see an indication of packet loss above 0%, or the program doesn't return after about five seconds, there is an issue with your network configuration, e.g., routing.

TCP Connection

Check to see if you can make a TCP connection to the Events API with netcat:

nc -v events.pagerduty.com 443

You should see the following message: Connection to events.pagerduty.com port 443 [tcp/https] succeeded!.

If the netcat command takes too long to print a message, or produces an error message indicating a timeout, connection refusal, etc., check your network's firewall/ACL and refer to Safelist IPs.

📘

BSD netcat

If you’re using BSD netcat, you can add flags -G 5 -w 1. This will make the utility exit one second after establishing a connection (-w 1) and give up and report timeout after five seconds of waiting to establish one (-G 5).

Testing TLS

PagerDuty Agent installs with its own copy of the public server certificate, which it uses when it connects to validate the connection to PagerDuty’s Events API.

🚧

Supported Protocols and the Server Certificate

PagerDuty supports the TLSv1.2 protocol. Older protocols (e.g., SSLv3 and TLSv1.0) are not supported due to security concerns. Traffic using older protocols and versions of TLS will be dropped.

For the security-conscious, the following command checks whether you’re able to establish a TLS connection:

openssl s_client -host events.pagerduty.com -port 443

You should see messages indicating a successful connection/TLS handshake, including the following certificate chain:

Certificate chain
 0 s:/OU=GT12858685/OU=See www.rapidssl.com/resources/cps (c)14/OU=Domain Control Validated - RapidSSL(R)/CN=*.pagerduty.com
   i:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA - G3
 1 s:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA - G3
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA

If you get a TLS handshake error, it may be the case that the GeoTrust root CA certificate isn't trusted on your system. You can find this certificate at DigiCert Trusted Root Authority Certificates.

In most distributions of Linux, the ca-certificates utility can be used for managing trusted certificates.

Proxy Troubleshooting

Issues with proxies can arise if the daemon’s environment variables are not properly set or if there's an issue connecting to the proxy server (e.g., LAN restrictions). For instructions on configuring PagerDuty Agent to use a proxy server, refer to the PagerDuty Agent installation guide.

Network Issues

Please try the following if you encounter networking issues:

Example
In the following example, we’ll assume the proxy server’s hostname is proxy.local and the port is 8118.

Try the following commands to uncover any networking issues:

1. ping proxy.local to see if it can be reached.
2. dig proxy.local or host proxy.local to see if the hostname resolves.
3. nc -v proxy.local 8118 to check that the port is open and accepts connections.
4. Try using curl to connect through the proxy to the Events API's agent heartbeat endpoint. Client authentication methods vary; please reference man curl for more information about any options specific to your environment. For example:

curl -proxy proxy.local:8118 [additional proxy options] https://api.pagerduty.com/agent/2014-03-14/heartbeat

If you’re able to successfully connect, you should receive a response similar to {"heartbeat_interval_secs":86400}.

Environment Variables

The agent daemon requires access to environment variables in order to use a proxy and send events to PagerDuty. This is because the underlying Python library used to make the HTTPS connection is urllib2. From the documentation on urllib2.urlopen:

In addition, if proxy settings are detected (for example, when a *_proxy environment variable like http_proxy is set), ProxyHandler is default installed and makes sure the requests are handled through the proxy.

Under Using a Proxy in the PagerDuty Agent Integration Guide, two ways to configure the environment variables for the daemon are described:

• systemd
• SysV

You can find out which of these your system uses by running sudo stat /proc/1/exe, which will show you the executable associated with the PID=1 process.

Once you've made sure that the daemon is configured with appropriate environment variables, you will need to make sure the changes are applied, which may vary based on your system. In systemd-based systems, you will be prompted to run systemctl daemon-reload when you restart pdagent, since a restart is required in order for changes to the service configuration to take effect.

Other Troubleshooting Steps

You may also wish to see the environment variables that the daemon uses by adding a logging statement to the agent's component module sendevent. On most Linux systems this will be in /lib/python2.7/site-packages/pdagent/. Add the following line to SendEventTask.send_event:

def send_event(self, json_event_str, event_id):
    import os; logger.info(str(os.environ)) # add this line
    ...

After making the change, restart pdagent and try sending a test event using pd-send. After that, you'll find a message in the log that that looks like this:

2017-05-08 18:29:31,556 INFO    SendEventTask        pdagent.sendevent    {'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'SHLVL': '2', 'SYSTEMCTL_SKIP_REDIRECT': '', 'PWD': '/', 'LOGNAME': 'pdagent', 'USER': 'pdagent', 'HOME': '/home/pdagent', 'SYSTEMCTL_IGNORE_DEPENDENCIES': '', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'XDG_SESSION_ID': '1', '_': '/usr/share/pdagent/bin/pdagentd.py'}

If the environment variables were set, they should show up in the print-out of the os.environ dictionary, i.e., 'http_proxy': 'http://proxy.local:8118' and 'https_proxy': 'https://proxy.local:8118', which are missing in the above example.