2

I have a problem with automatically mail notification in Nagios Core 4 installed on Ubuntu 12.04 LTS (Precise Pangolin) server...

I have tried to send mail with the nagios user and root user with the command:

echo "test" | mail -s "test mail" [email protected]

And I received the mail correctly... But I don't receive any automatically mail notification. How can I resolve this issue?

These are my configuration files (commands.cfg, contacts.cfg, nagios.log, mail.log):

commands.cfg

(The path /usr/bin/mail is the right path)

# 'notify-host-by-email' command definition
define command{
        command_name    notify-host-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\nHost: $HOSTNAME$\nState: $HOSTSTATE$\nAddress: $HOSTADDRESS$\nInfo: $HOSTOUTPUT$\n\nDate/Time: $LONGDATETIME$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Host Alert: $HOSTNAME$ is $HOSTSTATE$ **" $CONTACTEMAIL$
        }

# 'notify-service-by-email' command definition
define command{
        command_name    notify-service-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
        }
# 'process-host-perfdata' command definition
define command{
        command_name    process-host-perfdata
        command_line    /usr/bin/printf "%b" "$LASTHOSTCHECK$\t$HOSTNAME$\t$HOSTSTATE$\t$HOSTATTEMPT$\t$HOSTSTATETYPE$\t$HOSTEXECUTIONTIME$\t$HOSTOUTPUT$\t$HOSTPERFDATA$\n" >> /usr/local/nagios/var/host-perfdata.out
        }
# 'process-service-perfdata' command definition
define command{
        command_name    process-service-perfdata
        command_line    /usr/bin/printf "%b" "$LASTSERVICECHECK$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICESTATE$\t$SERVICEATTEMPT$\t$SERVICESTATETYPE$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$\n" >> /usr/local/nagios/var/service-perfdata.out
        }

contacts.cfg:

define contact{
        contact_name                    supporto
        alias                           Supporto Clienti DEA
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r
        host_notification_options       d,r
        service_notification_commands   notify-service-by-email
        host_notification_commands      notify-host-by-email
        email                           [email protected]
        }

define contactgroup{
        contactgroup_name       admins
        alias                   Nagios Administrators
        members                 supporto
        }

nagios.log:

[1401871412] SERVICE ALERT: fileserver;Current Users;OK;SOFT;2;USERS OK - 1 users currently logged in
[1401871953] SERVICE ALERT: backups;Nagios Status;WARNING;SOFT;1;NAGIOS WARNING: 36 processes, status log updated 541 seconds ago
[1401872133] SERVICE ALERT: backups;Nagios Status;OK;SOFT;2;NAGIOS OK: 36 processes, status log updated 180 seconds ago
[1401872321] SERVICE ALERT: posta;Swap Usage;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1401872322] SERVICE ALERT: fileserver;Current Users;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1401872420] SERVICE ALERT: archivio;Disk Space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1401872492] SERVICE ALERT: fileserver;Current Users;OK;SOFT;2;USERS OK - 1 users currently logged in
[1401872492] SERVICE ALERT: posta;Swap Usage;OK;SOFT;2;SWAP OK: 100% free (1984 MB out of 1984 MB)
[1401872590] SERVICE ALERT: archivio;Disk Space;OK;SOFT;2;DISK OK
[1401872931] Auto-save of retention data completed successfully.
[1401873333] SERVICE ALERT: backups;Nagios Status;WARNING;SOFT;1;NAGIOS WARNING: 36 processes, status log updated 402 seconds ago
[1401873513] SERVICE ALERT: backups;Nagios Status;OK;SOFT;2;NAGIOS OK: 36 processes, status log updated 180 seconds ago

mail.log

(I think that the problem is here, but I don't know how to resolve it.)

Jun  4 10:00:01 backups sm-msp-queue[6109]: My unqualified host name (backups) unknown; sleeping for retry
Jun  4 10:01:01 backups sm-msp-queue[6109]: unable to qualify my own domain name (backups) -- using short name
Jun  4 10:20:01 backups sm-msp-queue[7247]: My unqualified host name (backups) unknown; sleeping for retry
Jun  4 10:21:01 backups sm-msp-queue[7247]: unable to qualify my own domain name (backups) -- using short name
Jun  4 10:40:01 backups sm-msp-queue[8327]: My unqualified host name (backups) unknown; sleeping for retry
Jun  4 10:41:01 backups sm-msp-queue[8327]: unable to qualify my own domain name (backups) -- using short name
Jun  4 11:00:01 backups sm-msp-queue[9549]: My unqualified host name (backups) unknown; sleeping for retry
Jun  4 11:01:01 backups sm-msp-queue[9549]: unable to qualify my own domain name (backups) -- using short name
Jun  4 11:20:01 backups sm-msp-queue[10678]: My unqualified host name (backups) unknown; sleeping for retry
Jun  4 11:21:01 backups sm-msp-queue[10678]: unable to qualify my own domain name (backups) -- using short name

I'm at the last step, and I want to finish this Nagios Core! :)

Host definition (this host has the disk almost full, and it is in hard state but non notification):

define host{
        use                     generic-host            ; Name of host template to use
        host_name               posta
        alias                   Server Posta ESA
        address                 10.10.2.102
        parents                 xen1, xen2
        icon_image              redhat.png
        statusmap_image         redhat.gd2
        }

Service definition:

define service{
        use                             generic-service
        host_name                       xen1, maestro, xen2, posta, nas002, serv2, esasrvmi02, esaubuntumi
        service_description             Disk Space
        check_command                   ssh_all_disks!10%!5%
        }

Notification is allowed for the contact definition you gave, but is it also allowed at the the service level?

Sorry, but I don't understand this thing! :(

3 Answers 3

3

From your nagios.log I see only SOFT state errors. Nagios doesn't send any notification for SOFT state, only in case of HARD state. From the Nagios documentation :

Soft States

Soft states occur for services and hosts in the following situations...

1) When a service or host check results in a non-OK state and it has not yet been (re)checked the number of times specified by the option in the service or host definition. Let's call this a soft error state...

2) When a service or host recovers from a soft error state. This is considered to be a soft recovery.

Soft State Events

What happens when a service or host is in a soft error state or experiences a soft >recovery?

1) The soft error or recovery is logged if you enabled the log_service_retries or log_host_retries options in the main configuration file.

2) Event handlers are executed (if you defined any) to handle the soft error or recovery for the service or host. (Before any event handler is executed, the $STATETYPE$ macro is set to "SOFT"). Nagios does not send out notifications to any contacts because there is (or was) no "real" problem with the service or host.

As can be seen, the only important thing that really happens during a soft state is the execution of event handlers. Using event handlers can be particularly useful if you want to try and proactively fix a problem before it turns into a hard state.

Hard States

Hard states occur for hosts and services in the following situations:

1) When a host or service check results in a non-UP or non-OK state and it has been (re)checked the number of times specified by the max_check_attempts option in the host or service definition. This is a hard error state.

2) When a host or service transitions from one hard error state to another error state (e.g. WARNING to CRITICAL).

3) When a service check results in a non-OK state and its corresponding host is either DOWN or UNREACHABLE.

4) When a host or service recovers from a hard error state. This is considered to be a hard recovery.

5) When a passive host check is received. Passive host checks are treated as HARD unless the passive_host_checks_are_soft option is enabled.

The following things occur when hosts or services experience HARD state changes:

1) The HARD state is logged. 2) Event handlers are executed to handle the HARD state. 3) Contacts are notifified of the host or service problem or recovery.

So from what we see in the log you give in example, there was no need for Nagios to send a mail. You should create an error condition on one of the monitored services, let this condition exists for a while and see if you really get the mail when the state is changed to HARD in the nagios.log.

Last thing I've noticed, in your command line test, you send mail to [email protected] while in your contacts.cfg the mail address defined is [email protected] (maybe you have aliases defined on your mail servers maybe not).

Added after addition of logs to the question In the nagios.log that you show, there is no SERVICE NOTIFICATION line, so even when the error is in HARD state, Nagios doesn't even try to do the notification. To have notification working in Nagios, it is not sufficient to have the contacts, contact groups and notification commands well defined. You must configure per service and host if you want to send notification in case of error and of course to which contact(s) and / or contact group(s) to send this notification. For instance, this is a service definition with notification configured and working :

define service {
    name                                  generic-service
    first_notification_delay              0
    notification_interval                 0
    notification_options                  w,u,c,r
    notifications_enabled                 1
    check_period                          24x7
    notification_period                   24x7
    contact_groups                        admins
}

In the above definition, notification_enabled is set on 1 (true) and a contact group as been given to send notification to. Also, we define which kind of notification to send - w(arning), u(nknown), c(ritical) and r(ecovery).

The above definition is used as a template by all my services :

use generic-service

is present in all my services definitions. This way, if I need to change notification options, I just have to change the generic-service definition. In your case, your config shows that your service is using a template called generic-service. I would advice to check its definition to see if notification are configured like the example I gave above. Its definition can be located into a file called services-templates.cfg but this can vary.

7
  • oh sorry i forget a letter here in [email protected] the right mail address is supporto.. and in nagios there is the right mail address :)
    – alessio
    Jun 4, 2014 at 13:01
  • anyway how can i create an error condition for trying the mail notification? thanks in advance!
    – alessio
    Jun 4, 2014 at 13:05
  • If you monitor the disk space, define the error & warning thresolds lower than the space you currently have now for instance, just to see how your installation behace in case of error. Or stop a process which is monitored, ...
    – Benoit
    Jun 4, 2014 at 13:21
  • CURRENT SERVICE STATE: nas002;Disk Space;CRITICAL;HARD;4;DISK CRITICAL....
    – alessio
    Jun 4, 2014 at 13:36
  • but no notification! :( the mail.log it's ok?
    – alessio
    Jun 4, 2014 at 13:37
0

Thanks Benoit for your answer. I'd like to add a couple of things after fiddling with it for a while:

Easy to know where you stand with all those templates and overrides if you view the cache file, which is the computed result of all configurations: /usr/local/nagios/var/objects.cache

After I've been there it hit me.. my service was set to send notifications only in work hours, which ended up a bit wrong for me because I'm on a different time zone compared with the server. Changing it to 24x7 made it all work like a charm.

I hope this will help someone.. Spent hours figuring all this out.

Cheers!

0

The most helpful information was to croscheck the content of /usr/local/nagios/var/objects.cache to validate all inheritances of all .cfg's.

That was now also solving my issues.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .