Summary

PagerDuty SaaS

All PagerDuty systems exposed to the following CVEs have been addressed by upgrading to a patched version of log4j, either 2.16.0 or 2.12.2:

We currently see no evidence of compromises on our platform. Our teams continue to monitor for new developments and for impacts on sub-processors and dependent systems. PagerDuty SaaS customers do not need to take any additional action for their PagerDuty SaaS environment.

PagerDuty Security team is aware and has been responding to additional findings regarding vulnerabilities in the Log4j2 libraries identified in CVE-2021-45105. Where feasible, additional patching has occurred to further upgrade to 2.17.0.

Rundeck will release an updated version to address Log4J 2.17.0 on Tuesday, December 21st. We continue to strongly advise Rundeck customers to upgrade to the fully patched versions (3.3.16 or 3.4.8) immediately to secure and remediate the existing critical severity issues.

We continue to monitor the situation and provide similar remediations should they be necessary.

Rundeck Enterprise / Community

Rundeck 3.4.8 and 3.3.16 are available as of December 14th 2021 with the updated libraries necessary to bring Log4j up to a secure version. These versions will address the following two CVEs issued for the Log4j vulnerabilities:

For details of specific versions and mitigations, please see below.

Partial Mitigation Options for supported Rundeck vulnerable versions 3.3.7 - 3.4.7

NOTE: This partial mitigation will not fully address the vulnerability. It is strongly advised to upgrade to the fully patched versions (3.3.16 or 3.4.8) immediately to bring the libraries to a secure version.

If you are running Rundeck Enterprise or Rundeck Community, please immediately take the following actions to mitigate the CVE for Rundeck versions 3.4.x and 3.3.x to protect against the RCE vulnerability:

Add this flag to the JVM options for starting rundeck: -Dlog4j2.formatMsgNoLookups=true
Set env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Modify the file $RDECK_BASE/server/config/log4j2.properties, replace the string %m with %m{nolookups}

Rundeck Versions older than 1 year are not supported and should be immediately updated to a currently supported version. Please contact our support team ([email protected] or [email protected]) if you have any further questions.

December 15 - Final update

Our teams will continue to monitor for impacts on sub-processor or dependent systems, but we have updated all known areas of impact to our SaaS offering and we are continuing to monitor all PagerDuty services and environments. At this time PagerDuty SaaS customers do not need to take any additional action for their PagerDuty SaaS environment.

If you are running Rundeck Enterprise or Rundeck Community, you should apply the following patches to your environment immediately. Please see the important messages below.

Rundeck Enterprise versions 3.4.6 and below require you to immediately patch to mitigate the vulnerability.

Further details on the patch and mitigation are below and also on our webpage.

Rundeck Enterprise/Community

Rundeck 3.4.8 and 3.3.16 are available as of December 14th 2021 with the updated libraries necessary to bring Log4j up to a secure version. These versions will address both CVEs issued for the Log4j vulnerabilities.

CVE-2021-44228

Fixed in Rundeck 3.4.8/3.3.16
Partial fix in Rundeck 3.4.7/3.3.15
Partial mitigation in previous versions (3.4.6 and 3.3.14 and earlier) see section below.

CVE-2021-45046

Fixed in Rundeck 3.4.8/3.3.16

Partial Mitigation Options for supported Rundeck vulnerable versions 3.3.7 - 3.4.7

Add this flag to the JVM options for starting rundeck: -Dlog4j2.formatMsgNoLookups=true
Set env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Modify the file $RDECK_BASE/server/config/log4j2.properties, replace the string %m with %m{nolookups}

December 15th Update

On Friday December 10th 2021, PagerDuty became aware of a critical severity zero-day exploit known as “Log4Shell” in the Log4j library, which is widely used in numerous systems, websites, applications and digital services around the world. We reacted immediately and addressed the known issues across our portfolio in a patch release.

Subsequently, on Tuesday, December 14th 2021, a secondary CVE was published also relating to Log4j2. This necessitated a second patch release, which in some cases must be applied by our customers, specifically on customer managed products. Rundeck Enterprise and Community customers that have already patched to 3.4.7 should now also upgrade to 3.4.8 to address these new vulnerabilities.

If you are running Rundeck Enterprise or Rundeck Community, you should apply the following patches to your environment immediately. Please see the important messages below.

Rundeck Enterprise versions 3.4.6 and below require you to immediately patch to mitigate the vulnerability.

Further details on the patch and mitigation are below and also on our webpage.

Rundeck Enterprise/Community

CVE-2021-44228

Fixed in Rundeck 3.4.8/3.3.16
Partial fix in Rundeck 3.4.7/3.3.15
Partial mitigation in previous versions (3.4.6 and 3.3.14 and earlier) see section below.

CVE-2021-45046

Fixed in Rundeck 3.4.8/3.3.16

Partial Mitigation Options for Rundeck versions 3.4.6 and below

NOTE: This partial mitigation will not fully address the vulnerability. It is strongly advised to apply the above patches immediately to bring the libraries to a secure version.

If you are running Rundeck Enterprise or Rundeck Community, please immediately take the following actions to mitigate the CVE for Rundeck versions 3.4.6 and below to protect against the RCE vulnerability:

Add this flag to the JVM options for starting rundeck: -Dlog4j2.formatMsgNoLookups=true
Set env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Modify the file $RDECK_BASE/server/config/log4j2.properties, replace the string %m with %m{nolookups}

Please contact our support team ([email protected] or [email protected]) if you have any further questions.

December 13th

On Friday December 10th, PagerDuty and many others became aware of a critical severity zero-day exploit known as “Log4Shell” in the Log4j library, which is widely used in numerous systems around the internet. We immediately spun up a security incident and have been actively taking steps to mitigate and monitor the situation. We know our customers trust PagerDuty to manage their own incidents and we wanted to share a more detailed update of impact and progress to date.

PagerDuty’s SaaS platform does not make extensive use of Java. However in services and components where we identified use of this library, the vulnerability did not appear exploitable or we are running a current version that is not impacted by this exploit.

PagerDuty systems that leveraged the Log4j library version that required mitigation in order to avoid the log4j zero-day vulnerability “CVE-2021-44228” have been updated.

If you are running Rundeck Enterprise or Open Source, a patch update is required. Please see important messages below.

Where we did locate the vulnerable library, those services have already been updated out of an abundance of caution, and all known areas were upgraded by end of day Friday, December 10th.
PagerDuty’s network and firewall architecture limits the ability of this exploit to be successful, and we have added additional steps at our network layer to reduce the possibility of exploitation.
We continue to monitor for attempts by threat actors to attempt the exploit. If your organization is doing the same, you may consider creating an alert in PagerDuty.
Rundeck Enterprise versions 3.4.6 and below require you to immediately patch to mitigate the vulnerability. Further details on the patch and mitigation are below and also on our webpage.
We will continue to coordinate with our subprocessors, including AWS, to determine impacts to their environments and services as well, and we will continue to take action on any updates required as we learn of them.

PagerDuty SaaS

Rundeck Enterprise/OSS

If you are running Rundeck Enterprise or Open Source, please immediately take the following actions to mitigate the CVE for Rundeck versions 3.4.6 and below:

Add this flag to the JVM options for starting rundeck:
-Dlog4j2.formatMsgNoLookups=true
Set env var LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Modify the file $RDECK_BASE/server/config/log4j2.properties, replace the string %m with %m{nolookups}

Rundeck 3.4.7 GA is available as of December 10th with the updated libraries necessary to bring Log4j up to a safe version.

Rundeck 3.3.15 GA was released on December 13th.

Please contact our support team ([email protected] or [email protected]) if you have any further questions.