Custom Notification Data Sent from Nagios to PagerDuty

Follow

To customize the notification data Nagios sends to PagerDuty, you'll want to modify the commands specified in pagerduty_nagios.cfg. On Debian-based systems this file is usually found in /etc/nagios3/conf.d, and on RHEL-based systems you're most likely to find this file in /etc/nagios.

Looking at the command_line option, where pd-nagios is invoked (or pagerduty_nagios.pl, if you use the older Perl-based integration), you can add fields with the -f parameter, or modify existing fields as desired. For example, you may customize the pd_description with Nagios macros like $HOSTNAME$, $SERVICESTATE$, $LASTSERVICESTATE$, etc. as shown in the examples below. The information you include in pd_description will be relayed in the notifications sent to you via your personal notification rules.

Examples

Agent-based Integration

define command {
command_name notify-service-by-pagerduty
command_line /usr/share/pdagent-integrations/bin/pd-nagios -n service -k $CONTACTPAGER$ -t "$NOTIFICATIONTYPE$" -f SERVICEDESC="$SERVICEDESC$" -f SERVICESTATE="$SERVICESTATE$" -f HOSTNAME="$HOSTNAME$" -f HOSTDISPLAYNAME="$HOSTDISPLAYNAME$" -f SERVICEDISPLAYNAME="$SERVICEDISPLAYNAME$" -f SERVICEPROBLEMID="$SERVICEPROBLEMID$" -f SERVICEOUTPUT="$SERVICEOUTPUT$" -f pd_description="$SERVICEDESC$ : $SERVICEOUTPUT$"
}

define command {
command_name notify-host-by-pagerduty
command_line /usr/share/pdagent-integrations/bin/pd-nagios -n host -k $CONTACTPAGER$ -t "$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f HOSTSTATE="$HOSTSTATE$" -f HOSTDISPLAYNAME="$HOSTDISPLAYNAME$" -f HOSTPROBLEMID="$HOSTPROBLEMID$" -f pd_description="$SERVICEDESC$ : $SERVICEOUTPUT$"
}

Perl-based Integration

define command {
command_name     notify-service-by-pagerduty
command_line     /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=service -f pd_description="$SERVICEDESC$ : $SERVICEOUTPUT$"
}

define command {
command_name     notify-host-by-pagerduty
command_line     /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=host -f pd_description="$SERVICEDESC$ : $SERVICEOUTPUT$"
}

The Standard Macros in Nagios list provides full list of Nagios macros you can send to PagerDuty.

Have more questions? Submit a request

Comments

  • Avatar
    Moutten

    Are there any other attributes of a nagios alert that can be configured other than pd_nagios_object and pd_description?

  • Avatar
    Ryan Hoskin

    Hello,

    You can specify additional information that would normally be passed as an environment variable.  Here are some example commands:

    define command {

    command_name    notify-service-by-pagerduty

    command_line    /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=service -f CONTACTPAGER="$CONTACTPAGER$" -f NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f SERVICEDESC="$SERVICEDESC$" -f SERVICESTATE="$SERVICESTATE$"

    }

     

    define command {

    command_name    notify-host-by-pagerduty

    command_line    /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=host -f CONTACTPAGER="$CONTACTPAGER$" -f NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f HOSTSTATE="$HOSTSTATE$"

    }

  • Avatar
    Tobias Schmidt

    Hey,

    I was able to add the actual nagios service output by adding -f SERVICEDESC="$SERVICEDISPLAYNAME$: $SERVICEOUTPUT$" to the enqueue command. Though, is there a way to keep the SERVICE part in an incident view, but add a DESCRIPTION part with the actual SERVICEOUTPUT?

    My main issue with the default nagios setup is that the SERVICEOUTPUT is hidden in the long list of all details. Ideally, I want to have HOST, SERVICE, DESCRIPTION, STATE and DETAILS as sections. Overwriting SERVICE is a workaround though.

    Is there a documentation about what ENVs end up in separate sections in an incident view?

    Thanks!

  • Avatar
    Tobias Schmidt

    Oh, by the way, pd_description had no effect at all it seems. How is it supposed to work?

  • Avatar
    Ryan Hoskin

    For a PagerDuty service Nagios incidents will be de-duped based on the $HOSTNAME$ and $SERVICEDESC$ (if it's a service). You can also specify your own key for de-duplication in the pd_incident_key field. Below is a sample switch to generate the key for each incident:

    -f pd_incident_key="$HOSTNAME$ $SERVICEDESC$"

    $HOSTNAME$ and $SERVICEDESC$ could be replaced with any of the Nagios macros ( http://nagios.sourceforge.net/docs/3_0/macrolist.html).

    The pd_description field unfortunately won't make the incidents table any more useful. You could adjust the SERVICEDESC and then use the pd_incident_key as described above.

  • Avatar
    Ophir Ronen

    I'm working on enriching the events from Nagios to include escalation and remediation information for the benefit of the NOC / on-call teams. Where I was running into a problem was getting that information into PagerDuty incidents without all of the associated noise of the 200+ Nagios macros that were coming along for the ride. :)

    I dug into the integration and found that it is actually pretty straightforward to customize the incident generated by the Nagios event. My findings are documented in the following post: 

    http://www.eventenrichment.org/integrate-nagios-pagerduty-engage-operations-nervous-system/

    In short, a minor modification to pagerduty_nagios.pl and a number of modifications to pagerduty_nagios.cfg and you are good to go. :)

  • Avatar
    Ryan Hoskin

    That was a very nice addition to the integration Ophir.  Thank you for working through that and providing it for others.

  • Avatar
    Zach Drew

    I attempted to follow the instructions from Ophir, in the most recent version, the section to be commented out starts at line 219, not 277 as stated.

    when I commented out the section, my alerts stopped getting to pagerduty.

     

    I am not a perl guy, but it seems to me that the default configuration of drinking from the firehose creates a lot of noise.  I understand why it is the way it is, but I think there should be an easier option for those that want to use a custom alert string.

     

  • Avatar
    Ryan Hoskin

    Your line number is correct Zach.  Can you contact us at support@pagerduty.com so that we can help you troubleshoot?

  • Avatar
    Ryan Hoskin

    To all future readers of this article.  If you're going to follow Ophir's guide, it's probably easier for you to modify enable_environment_macros=0 in your nagios.cfg than it is to comment out a portion of perl script.  They both should achieve the same functionality.  The only other thing you would need to do besides disabling the environment macros is to change the commands for notify-host-by-pagerduty and notify-service-by-pagerduty.

  • Avatar
    Ophir Ronen

    Thanks Ryan, that is much easier!  As Leonardo Da Vinchi was purported to have said: “Simplicity is the ultimate sophistication. " :)

    I tested this out in the lab and it works just fine with enable_environment_macros set to 0 and no modifications to pagerduty_nagios.pl.

    The guide is up to date: http://www.eventenrichment.org/integrate-nagios-pagerduty-engage-operations-nervous-system/ 

     

     

  • Avatar
    Bryan D Chapman

    This is just what we want - so close but I cant get it working!

    After setting enable_environment_macros=0 and creating a new pagerduty_nagios.cfg file I get the following in /var/log/messages

    Mar 5 14:50:43 huginn pagerduty_nagios[5234]: Nagios event in file /tmp/pagerduty_nagios/pd_1394031043_5234.txt REJECTED by the PagerDuty server. Server says: The NOTIFICATIONTYPE field must be present and must be one of: PROBLEM, ACKNOWLEDGEMENT, RECOVERY, NOP. The following fields must be present when pd_nagios_object = service: HOSTNAME, SERVICEDESC, SERVICESTATE

  • Avatar
    Ryan Hoskin

    Hi Bryan,

    That sounds like an issue with your environment variables.  Would you please check that they are enabled and then email support@pagerduty.com with your Nagios version?

    Thanks,

    Ryan

  • Avatar
    Trever Furnish

    Whoa there, buddy!

    Setting enable_environment_macros=0 is cutting out key Nagios functionality needed in most event handlers. That's definitely not a workable approach for my installations. (And I'm aka tgfurnish@herffjones.com, long-time customer.)

    I agree with the sentiment that a more straightforward and supportable approach is needed.

  • Avatar
    Ryan Hoskin

    Hi Trever,

    I just shot you an email in this regard. Apologies for the delay.

    Cheers,
    Ryan