--- updatedAt: 2026-06-11T18:17:39.000Z agentTools: siteIndex: https://support.pagerduty.com/llms.txt projectIndex: https://support.pagerduty.com/main/llms.txt --- # PagerDuty Agent Troubleshooting Guide [PagerDuty Agent](https://github.com/PagerDuty/pdagent) is an open-source utility for integrating standalone/on-premise monitoring tools with PagerDuty. This guide offers help for troubleshooting the most common issues. * [Basic Troubleshooting](#basic-troubleshooting) * [Outqueue Troubleshooting](#outqueue-troubleshooting) * [Network Troubleshooting](#network-troubleshooting) * [Testing TLS](#testing-tls) * [Proxy Troubleshooting](#proxy-troubleshooting) > 📘 Looking for PagerDuty AI Agents? > > PagerDuty Agent is an application that integrates with traditional monitoring software. If you're looking for our AI-driven agents, please read [PagerDuty Advance](https://support.pagerduty.com/main/docs/pagerduty-advance#ai-agents) for more information. # Basic Troubleshooting If you have trouble sending events to PagerDuty using the agent, run the following checks first: * **[Verify that the `pdagent` service is running.](https://support.pagerduty.com/main/docs/pagerduty-agent-integration-guide#verify-the-agent-is-running)** If the service is not running, you can start it with the command `systemctl start pdagent` (on systemd-based systems) or `sudo service pdagent start` (on SysV init-based systems). Any enqueued events that accumulated while the service was stopped will be sent when the service starts again. * **Check the service integration key**: Make sure that the key is for the correct service and that it has been entered correctly. * **[Send a test event using `pd-send`](https://www.pagerduty.com/docs/guides/agent-install-guide#pd-send)**. Remember to replace the key used in the example with your own PagerDuty service integration key. * **Check the logs** for any errors, including events that could not be submitted to the Events API. These are typically found under `/var/log/pdagent/pdagentd.log`. ## Installation Error: `pdagent.service not found` Sometimes on a fresh install of the PagerDuty Agent, you might see the error message `Failed to start pdagent.service: Unit pdagent.service not found.` This is due to an error during installation where the file `pdagent.service` was not copied to the proper directory. To fix this, copy the `pdagent.service` file into the proper directory: 1. Find the file `pdagent.service` with the command: ```shell sudo find / -name pdagent.service ``` 2. Copy the location of the file. 3. Copy the file to the correct directory with the command: ```shell sudo cp /etc/systemd/system ``` 4. Restart the agent with `systemctl start pdagent` or `service pdagent start`, depending on whether your operating system uses `systemctl` or `service`. # Outqueue Troubleshooting In addition to [using `pd-queue` to manage the queue](https://support.pagerduty.com/main/docs/pagerduty-agent-integration-guide#pd-queue--h), the agent's [outqueue directories](#outqueue-directories) often contain useful troubleshooting information. You can also [manually clear the queue](#manually-purge-the-queue) by working directly with the files. Each file in the [outqueue directories](#outqueue-directories) contains the body of an HTTP POST request that will be made to the [PagerDuty Events API](https://developer.pagerduty.com/api-reference/YXBpOjI3NDgyNjU-pager-duty-v2-events-api). You can examine these files to see what your monitoring tools are submitting to the agent. File names follow the naming convention `{UNIX timestamp}_{integration key}.txt`. ## Outqueue Directories The queue directories, located in `/var/lib/pdagent/outqueue`, include: | Directory | Details | | :-------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `err` | Events that `pdagent` attempts to deliver to the PagerDuty Events API but received a `400 Invalid Request` response from PagerDuty will be logged here. Malformed events — for example, missing an integration key — that cannot be submitted are kept here as well. | | `suc` | Successfully submitted events. If you see an event file cached here, it was sent to PagerDuty. If the event did not trigger an incident in PagerDuty and you are having difficulty troubleshooting, refer to [Why Incidents Fail to Trigger](https://support.pagerduty.com/main/docs/why-incidents-fail-to-trigger). | | `tmp` | Event files submitted to the agent by external processes, such as integration scripts or `pd-send`, will be stored in this directory. | | `pdq` | The queue of events to be sent; contains a symbolic link to the files in `tmp`. | > ⚠️ Event Expiration and Queue Cleaning > > Events are deleted from the outqueue directories at a regular interval once their age exceeds a predefined length, in seconds. The default age threshold is one week, and the default cleaning interval is three hours. Both the interval and the age threshold are defined in the [PagerDuty Agent configuration file](https://github.com/PagerDuty/pdagent/blob/master/conf/pdagent.conf), which should be located at `/etc/pdagent.conf`: > > * `cleanup_interval_secs`: The interval between when the queue is cleaned. > * `cleanup_threshold_secs`: The age, in seconds, after which events will be deleted. ## Manually Purge the Queue In the event of a long network outage where a large number of events accumulate in the agent's outqueue, you can manually clear out the queue to avoid a heavy cluster of PagerDuty incidents when network connectivity is restored. To clear the queue, run the following commands: ```shell sudo rm -vf /var/lib/pdagent/outqueue/pdq/* sudo rm -vf /var/lib/pdagent/outqueue/tmp/* ``` # Network Troubleshooting The host running PagerDuty Agent should be able to make an HTTPS connection to `events.pagerduty.com`. If the LAN does not have DNS service, there is no route to the internet, or the local ACL does not permit connecting to remote hosts, the agent will not be able to send data to PagerDuty. The following items can help resolve network issues: * [DNS](#dns) * [Reachability](#reachability) * [TCP Connection](#tcp-connection) ## DNS In the host's command line, run the following command: ```shell dig events.pagerduty.com ``` If you see a status response of `SRVFAIL` or `NXDOMAIN`, the issue is most likely with DNS. ## TCP Connection Check whether you can make a TCP connection to the Events API with [netcat](http://netcat.sourceforge.net/): ```shell nc -v events.pagerduty.com 443 ``` You should see the following message: `Connection to events.pagerduty.com port 443 [tcp/https] succeeded!`. If the netcat command takes too long to print a message, or produces an error indicating a timeout, connection refusal, or similar issue, check your network's firewall/ACL and refer to [Safelist IPs](https://support.pagerduty.com/main/docs/safelist-ips). > 📘 BSD netcat > > If you are using BSD netcat, you can add flags `-G 5 -w 1`. This will make the utility exit one second after establishing a connection (`-w 1`) and give up and report timeout after five seconds of waiting to establish one (`-G 5`). # Testing TLS PagerDuty Agent installs with its own copy of the public server certificate, which it uses when it connects to validate the connection to PagerDuty's [Events API](https://developer.pagerduty.com/api-reference/YXBpOjI3NDgyNjU-pager-duty-v2-events-api). > ⚠️ Supported Protocols and the Server Certificate > > PagerDuty supports the TLSv1.2 protocol. Older protocols such as SSLv3 and TLSv1.0 are not supported due to security concerns. Traffic using older protocols and versions of TLS will be dropped. The following command checks whether you are able to establish a TLS connection: ```shell openssl s_client -host events.pagerduty.com -port 443 ``` You should see messages indicating a successful connection and TLS handshake, including the following certificate chain: ```shell Certificate chain 0 s:/OU=GT12858685/OU=See www.rapidssl.com/resources/cps (c)14/OU=Domain Control Validated - RapidSSL(R)/CN=*.pagerduty.com i:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA - G3 1 s:/C=US/O=GeoTrust Inc./CN=RapidSSL SHA256 CA - G3 i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA ``` If you get a TLS handshake error, it may be that the GeoTrust root CA certificate is not trusted on your system. You can find this certificate at [DigiCert Trusted Root Authority Certificates](https://www.digicert.com/kb/digicert-root-certificates.htm). In most distributions of Linux, the `ca-certificates` utility can be used for managing trusted certificates. # Proxy Troubleshooting Issues with proxies can arise if the daemon's environment variables are not properly set or if there is an issue connecting to the proxy server, such as LAN restrictions. For instructions on configuring PagerDuty Agent to use a proxy server, refer to the [PagerDuty Agent installation guide](https://support.pagerduty.com/main/docs/pagerduty-agent-integration-guide#using-a-proxy). ## Network Issues Try the following if you encounter networking issues. **Example** In the following example, the proxy server's hostname is `proxy.local` and the port is `8118`. Try the following commands to uncover any networking issues: 1. `ping proxy.local` to see if it can be reached. 2. `dig proxy.local` or `host proxy.local` to see if the hostname resolves. 3. `nc -v proxy.local 8118` to check that the port is open and accepts connections. 4. Use `curl` to connect through the proxy to the Events API's agent heartbeat endpoint. Client authentication methods vary — reference `man curl` for more information about options specific to your environment. For example: ```shell curl -proxy proxy.local:8118 [additional proxy options] https://api.pagerduty.com/agent/2014-03-14/heartbeat ``` If you are able to successfully connect, you should receive a response similar to `{"heartbeat_interval_secs":86400}`. ## Environment Variables The agent daemon requires access to environment variables to use a proxy and send events to PagerDuty. This is because the underlying Python library used to make the HTTPS connection is `urllib2`. From the [documentation on urllib2.urlopen](https://docs.python.org/2/library/urllib2.html#urllib2.urlopen): > In addition, if proxy settings are detected (for example, when a `*_proxy` environment variable like `http_proxy` is set), [`ProxyHandler`](https://docs.python.org/2/library/urllib2.html#urllib2.ProxyHandler) is default installed and makes sure the requests are handled through the proxy. Under **Using a Proxy** in the [PagerDuty Agent Integration Guide](https://support.pagerduty.com/main/docs/pagerduty-agent-integration-guide#using-a-proxy), two ways to configure the environment variables for the daemon are described: * [systemd](https://support.pagerduty.com/main/docs/pagerduty-agent-integration-guide#systemd-based-systems) * [SysV](https://support.pagerduty.com/main/docs/pagerduty-agent-integration-guide#sysv-init-based-systems) You can find out which of these your system uses by running `sudo stat /proc/1/exe`, which will show you the executable associated with the PID=1 process. Once you have confirmed that the daemon is configured with appropriate environment variables, make sure the changes are applied — this may vary based on your system. In systemd-based systems, you will be prompted to run `systemctl daemon-reload` when you restart `pdagent`, since a restart is required for changes to the service configuration to take effect. ## Other Troubleshooting Steps You may also wish to see the environment variables that the daemon uses by adding a logging statement to the agent's component module [`sendevent`](https://github.com/PagerDuty/pdagent/blob/master/pdagent/sendevent.py). On most Linux systems this will be in `/lib/python2.7/site-packages/pdagent/`. Add the following line to `SendEventTask.send_event`: ```python def send_event(self, json_event_str, event_id): import os; logger.info(str(os.environ)) # add this line ... ``` After making the change, restart `pdagent` and try sending a test event using `pd-send`. After that, you will find a message in the log that looks like this: ```text 2017-05-08 18:29:31,556 INFO SendEventTask pdagent.sendevent {'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'SHLVL': '2', 'SYSTEMCTL_SKIP_REDIRECT': '', 'PWD': '/', 'LOGNAME': 'pdagent', 'USER': 'pdagent', 'HOME': '/home/pdagent', 'SYSTEMCTL_IGNORE_DEPENDENCIES': '', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'XDG_SESSION_ID': '1', '_': '/usr/share/pdagent/bin/pdagentd.py'} ``` If the environment variables were set, they should appear in the print-out of the `os.environ` dictionary — for example, `'http_proxy': 'http://proxy.local:8118'` and `'https_proxy': 'https://proxy.local:8118'` — which are missing in the above example.