Summary

PagerDuty SaaS

All PagerDuty systems exposed to the following CVEs have been addressed by upgrading to a patched version of log4j — either 2.16.0 or 2.12.2:

We currently see no evidence of compromise on our platform. Our teams continue to monitor for new developments and for impacts on sub-processors and dependent systems. PagerDuty SaaS customers do not need to take any additional action for their PagerDuty SaaS environment.

The PagerDuty Security team is aware of and has been responding to additional findings regarding vulnerabilities in the Log4j2 libraries identified in CVE-2021-45105. Where feasible, additional patching has occurred to upgrade to 2.17.0.

Rundeck will release an updated version to address Log4j 2.17.0 on Tuesday, December 21st. We continue to strongly advise Rundeck customers to upgrade to the fully patched versions (3.3.16 or 3.4.8) immediately to secure and remediate the existing critical severity issues.

We continue to monitor the situation and will provide similar remediations should they be necessary.

Rundeck Enterprise / Community

Rundeck 3.4.8 and 3.3.16 are available as of December 14th, 2021 with the updated libraries necessary to bring Log4j to a secure version. These versions address the following CVEs:

For details on specific versions and mitigations, see below.

Partial Mitigation Options for Supported Rundeck Versions 3.3.7–3.4.7

🚧
Partial Mitigation Only
This partial mitigation does not fully address the vulnerability. Upgrade to the fully patched versions (3.3.16 or 3.4.8) immediately to bring the libraries to a secure version.

If you are running Rundeck Enterprise or Rundeck Community versions 3.4.x or 3.3.x, immediately take the following actions to mitigate the RCE vulnerability:

Add this flag to the JVM options for starting Rundeck: -Dlog4j2.formatMsgNoLookups=true
Set env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Modify the file $RDECK_BASE/server/config/log4j2.properties and replace the string %m with %m{nolookups}

Rundeck versions older than one year are not supported and must be updated to a currently supported version immediately. Contact our support team at [email protected] or [email protected] with any further questions.

December 15 — Final Update

Our teams will continue to monitor for impacts on sub-processor or dependent systems. We have updated all known areas of impact to our SaaS offering and continue to monitor all PagerDuty services and environments. PagerDuty SaaS customers do not need to take any additional action for their PagerDuty SaaS environment.

If you are running Rundeck Enterprise or Rundeck Community, apply the following patches to your environment immediately.

Rundeck Enterprise versions 3.4.6 and below require immediate patching to mitigate the vulnerability.

Further details on the patch and mitigation are available below and on the Rundeck CVE page.

Rundeck Enterprise / Community

Rundeck 3.4.8 and 3.3.16 are available as of December 14th, 2021 with the updated libraries necessary to bring Log4j to a secure version. These versions address both CVEs issued for the Log4j vulnerabilities.

CVE-2021-44228

Fixed in Rundeck 3.4.8 / 3.3.16
Partial fix in Rundeck 3.4.7 / 3.3.15
Partial mitigation in previous versions (3.4.6 and 3.3.14 and earlier) — see the section below

CVE-2021-45046

Fixed in Rundeck 3.4.8 / 3.3.16

Partial Mitigation Options for Supported Rundeck Versions 3.3.7–3.4.7

🚧
Partial Mitigation Only
This partial mitigation does not fully address the vulnerability. Upgrade to the fully patched versions (3.3.16 or 3.4.8) immediately to bring the libraries to a secure version.

If you are running Rundeck Enterprise or Rundeck Community versions 3.4.x or 3.3.x, immediately take the following actions to mitigate the RCE vulnerability:

Add this flag to the JVM options for starting Rundeck: -Dlog4j2.formatMsgNoLookups=true
Set env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Modify the file $RDECK_BASE/server/config/log4j2.properties and replace the string %m with %m{nolookups}

December 15 — Update

On Friday, December 10th, 2021, PagerDuty became aware of a critical severity zero-day exploit known as "Log4Shell" in the Log4j library, which is widely used in numerous systems, websites, applications, and digital services around the world. We reacted immediately and addressed the known issues across our portfolio in a patch release.

Subsequently, on Tuesday, December 14th, 2021, a secondary CVE was published relating to Log4j2. This required a second patch release, which in some cases must be applied by customers on customer-managed products. Rundeck Enterprise and Community customers who have already patched to 3.4.7 must also upgrade to 3.4.8 to address these new vulnerabilities.

If you are running Rundeck Enterprise or Rundeck Community, apply the following patches to your environment immediately.

Rundeck Enterprise versions 3.4.6 and below require immediate patching to mitigate the vulnerability.

Further details on the patch and mitigation are available below and on the Rundeck CVE page.

Rundeck Enterprise / Community

CVE-2021-44228

Fixed in Rundeck 3.4.8 / 3.3.16
Partial fix in Rundeck 3.4.7 / 3.3.15
Partial mitigation in previous versions (3.4.6 and 3.3.14 and earlier) — see the section below

CVE-2021-45046

Fixed in Rundeck 3.4.8 / 3.3.16

Partial Mitigation Options for Rundeck Versions 3.4.6 and Below

🚧
Partial Mitigation Only
This partial mitigation does not fully address the vulnerability. Apply the patches above immediately to bring the libraries to a secure version.

If you are running Rundeck Enterprise or Rundeck Community versions 3.4.6 and below, immediately take the following actions to mitigate the RCE vulnerability:

Add this flag to the JVM options for starting Rundeck: -Dlog4j2.formatMsgNoLookups=true
Set env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Modify the file $RDECK_BASE/server/config/log4j2.properties and replace the string %m with %m{nolookups}

Contact our support team at [email protected] or [email protected] with any further questions.

December 13

On Friday, December 10th, PagerDuty and many others became aware of a critical severity zero-day exploit known as "Log4Shell" in the Log4j library, which is widely used in numerous systems around the internet. We immediately initiated a security incident and have been actively taking steps to mitigate and monitor the situation. We know our customers trust PagerDuty to manage their own incidents, and we want to share a more detailed update on impact and progress to date.

PagerDuty's SaaS platform does not make extensive use of Java. In the services and components where we identified use of this library, the vulnerability did not appear exploitable, or we are running a current version not impacted by this exploit.

PagerDuty systems that leveraged the vulnerable version of the Log4j library required to mitigate CVE-2021-44228 have been updated.

⚠️
Rundeck Enterprise and Open Source Customers
If you are running Rundeck Enterprise or Open Source, a patch update is required. See the important messages below.

Where we identified the vulnerable library, those services have already been updated, and all known areas were upgraded by end of day Friday, December 10th.
PagerDuty's network and firewall architecture limits the ability of this exploit to succeed, and we have added additional steps at our network layer to reduce the possibility of exploitation.
We continue to monitor for attempts by threat actors to exploit this vulnerability. If your organization is doing the same, you may consider creating an alert in PagerDuty.
Rundeck Enterprise versions 3.4.6 and below require immediate patching to mitigate the vulnerability. Further details on the patch and mitigation are available below and on the Rundeck CVE page.
We will continue to coordinate with our sub-processors, including AWS, to determine impacts to their environments and services, and will continue to take action on any updates required as we learn of them.

PagerDuty SaaS

Rundeck Enterprise / OSS

If you are running Rundeck Enterprise or Open Source versions 3.4.6 and below, immediately take the following actions to mitigate the CVE:

Add this flag to the JVM options for starting Rundeck: -Dlog4j2.formatMsgNoLookups=true
Set env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Modify the file $RDECK_BASE/server/config/log4j2.properties and replace the string %m with %m{nolookups}

Rundeck 3.4.7 GA is available as of December 10th with the updated libraries necessary to bring Log4j to a safe version:

Rundeck 3.3.15 GA was released on December 13th.

Contact our support team at [email protected] or [email protected] with any further questions.