Nagios XI error "Last Check Time Not Updating" occurs due to different reasons ranging from connection issue to the backend historical database, crashed database tables, core scheduling/check execution issues, or lack of resources.

Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to solve Nagios related issues.

In this context, we shall discuss this error and how to fix it.

What triggers Nagios XI error "Last Check Time Not Updating"?

As earlier stated, this issue is caused by different factors such as connection issues to the backend historical database, crashed database tables, core scheduling and other issues.

How to troubleshooting Nagios XI error "Last Check Time Not Updating"?

To troubleshoot this problem, you need to verify if the checks are actually getting scheduled and executed. In cases where they are not getting executed, then the problem is from Nagios Core engine.

Similarly, if the check are getting scheduled, the the issue is from the database.

To verify this is to check the Nagios Core web frontend to see if the "Last Check" time is updating. To do this, simply Browse to the following URL as per your Nagios server:

http://<server_ip_or_hostname>/nagios/

If the core interface is currently experiencing issues with with "Last Check" times then check any of the details for an object. Similarly, the Core interface displays accurate "Last Check" times, then head to Step 2 shown below. If the Core interface is experiencing the same issues as the XI interface then follow Step 1 below.

Step 1: The check is failing to be scheduled or executed

Issues with the Nagios Core auto-rescheduler directives:

To begin, with the introduction of the auto_rescheduling feature in Nagios Core 4.0.8 there were a few bugs. Those affected by this bug will notice the nagios.log file with errors relating to rescheduled checks.

By default, the new directives added to nagios.cfg could cause rescheduled checks to never execute, and instead be continuously rescheduled. The original /usr/local/nagios/etc/nagios.cfg directives were:

auto_reschedule_checks=1
auto_rescheduling_interval=30
auto_rescheduling_window=180

To fix this issue, reduce the auto_rescheduling_window to 45.

auto_reschedule_checks=1
auto_rescheduling_interval=30
auto_rescheduling_window=45

After implementing the above changes in the nagios.cfg file, then restart Nagios Core by running the following commands below:

For RHEL 6|CentOS 6|Oracle Linux 6|Ubuntu 14, execute;

service nagios restart

For RHEL 7|CentOS 7|Oracle Linux 7|Debian|Ubuntu 16/18, execute;

systemctl restart nagios.service

Resource Issues forcing the rescheduling of checks:

If the system "ulimit" settings are too restrictive, checks may be orphaned and forced to reschedule. Normally, this behavior is identified by checking the "nagios.log" file for lines similar to:

[1331905537] Warning: The check of service ‘SERVICE’ on host ‘NAMESERVER’ looks like it WAS orphaned (results never Came
back). I’m scheduling an immediate check of the service … [1331755699] Warning: The check of service ‘SWAP’ on host ‘nameserver’
not could be due to Performed to fork () error ‘Resource temporarily unavailable’. The check will be rescheduled.

If many of these lines exist in "nagios.log", the perform the following tasks to increase the kernel "ulimts":

Mpdify the file "/etc/security/limits.conf" and define/update the following settings:

#locked memory
* hard memlock 128
* soft memlock 128
#open files
* soft nofile 10000
* hard nofile 10000
root hard nofile 10000
root soft nofile 10000
#max user processes
* hard nproc 4096
* soft nproc 4096
#stack size
* hard stack 20480
* soft stack 20480

If the setting does not exist then add the line. After making the changes save the file and restart the server.

Once the reboot completes, run the following command to verify that the new settings are in place:

ulimit -a

Step 2. ndo2db is failing to insert the check result into the "Nagios" MySQL database.

Availability of crashed tables in the Nagios database:

You will see crashed tables located the MySQL/MariaDB logs as shown below:

/var/log/mysqld.log

or for MariaDB:

/var/log/mariadb/

The relevant errors should look like:

141127 10:40:24 [ERROR] /usr/libexec/mysqld: Table ‘./nagios/nagios_logentries’ is marked as crashed and last (automatic?) repair failed

Therefore, run the following commands to repair the tables.

cd /usr/local/nagiosxi/scripts/
./repair_databases.sh

After completing the above steps, make sure that multiple Nagios processes are not running. To check this status, run the following command;

ps -ef | grep nagios.cfg | grep -v grep

The following output is healthy:

nagios 5713 1 0 08:40 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 5723 5713 0 08:40 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg

You can see from the above output that there is only one PID 5713 which is the parent process.

The second line has the PID 5723. However you can see that it references the parent PID of 5713, this is a child process of the parent and is normal behavior. On heavily-loaded systems, you may see multiple child processes – this is also normal behavior.

In cases where the output has more than one parent process, use the following commands:

For RHEL 6|CentOS 6|Oracle Linux 6|Ubuntu 14, execute;

service nagios stop
killall -9 nagios
service nagios start

For RHEL 7|CentOS 7|Oracle Linux 7|Debian|Ubuntu 16/18, execute;

systemctl stop nagios.service
killall -9 nagios
systemctl start nagios.service

[Need support in fixing Nagios errors? We are available to help you today.]