Generally, it is easy to create alerts in Nagios Log Server and have them sent to a Nagios monitoring server with NRDP, Nagios Reactor to automate alert responses, send as a SNMP trap or simply send as an alert email.

Here at Ibmi Media, as part of our Server Management Services, we regularly help our Customers to perform related Nagios Log Server queries.

In this context, we shall look into create Nagios Log Server alerts.

Alerting On Log Events With Nagios Log Server

Alert Types

Three types of alerts in Nagios Log Server include:

• Query
• Real-Time
• Host Freshness

Query

i. These are based on the results of a query that has already been defined (located in the Dashboard menu), hence we will need to have a query defined before creating an alert.

ii. With these alerts, data is queried on an interval (usually five minutes) and is checked for any abnormalities. This means that, for a critical issue, alerts may be delayed by up to that check interval.

Real-Time

i. Real-time alerts are a way to circumvent the delay associated with interval-based queries. Instead, they exist in the Logstash configuration itself, checking each event as it comes in for that “abnormal” criteria.

ii. This feature should be used sparingly, as too many Logstash filters may degrade performance. However, for certain critical events, this may be worth the cost.

Host Freshness

Alert based on previously configured hosts where log data is no longer being received by Nagios Log Server instances from these hosts.

Alerting on Log Events with Nagios Log Server

In Nagios Log Server, select Alerting from the navigation bar.

This is the central location to manage and create alerts. We can also create alerts from the Dashboards menu, they will appear here once created.

There are multiple alert methods available in Nagios Log Server.

i. Nagios/NRDP – Send an alert to our Nagios XI or Nagios Core server using NRDP

ii. Execute Script – Run a custom script and pass variables to the script

iii. SNMP Traps – SNMP Traps can be sent to other applications using the Nagios MIB

iv. Email Users – Email Nagios Log Server users

v. Nagios XI Log Server Wizard – We can use the Nagios XI Log Server Wizard to alert based on queries saved on our Nagios Log Server

Certain alerts methods require us to define the settings (such as the NRDP server) before we can create an alert.

NRDP

Alerts can be sent to a Nagios XI or Nagios Core server running NRDP. Nagios XI comes pre-installed with NRDP, all that is required is to configure the token we wish to use. If we are using Nagios Core, we will need to first install and then configure NRDP.

Please take note of the NRDP Token we define as we will need it later.

In Nagios Log Server, in the left pane under Alert Settings click Nagios/NRDP, then click the Add NRDP Server button.

We will need to provide the following information:

i. Name: The name of the NRDP server we are adding.

ii. NRDP Address: The address of the Nagios server NRDP is configured for (we must include the http:// part of the URL).

iii. NRDP Token: Provide the Token we defined on our Nagios XI or Nagios Core server.

Click the Add button to define the NRDP server.

Execute Script

Nagios Log server allows us to execute a script as an alerting method. We will need to make sure that the script exists on all instances in our cluster. The script is executed on the master node of our cluster, this can change at any time to any instance in the cluster, hence why the script needs to be located on all instances.

SNMP Trap Receivers

To be able to send alerts to an SNMP Trap receiver, we need to define the details of the trap receiver. In Nagios Log Server, in the left pane under Alert Settings, click SNMP Trap Receivers, then click the Add SNMP Trap Receiver button.

We will need to provide the following information:

i. Name : The name of the SNMP Trap receiver we are adding.

ii. Receiver Address: The address that is receiving traps. Could be an

NSTI server or a Nagios XI server that is listening for incoming traps.

We also need to define the port the traps can be sent on (162 is the standard default).

iii. SNMP Version: The version of SNMP we are using, changing the version will change the trap security options available.

Version 2c

iv. Community String: The community string that the SNMP Trap receiver will accept traps for. This is commonly public but depends on how our SNMP Trap receiver is configured.

Version 3

v. Authorization Level: The authorization method used to send SNMP v3 traps. Our selection here defines the relevant Authorization and Privacy fields that are shown.

Click the Add button to define the SNMP Trap Receiver.

Email Users

To be able to send email alerts in Nagios Log Server, we will need to create Nagios Log Server user accounts with their email addresses correctly defined.

In Nagios Log Server navigate to Admin > Management > User Management.

The User Management page allows administrators to create new users and edit or delete existing users.

To create a new user, click the Create User button. We will be presented with a list of fields to populate for creating a new user, these are spread across several tabs starting with the Details tab:

a. User Details:

i. Full Name

ii. Email address

b. Account Information

i. Username

ii. Password

iii. Language

External Authentication: Nagios Log Server allows us to use external authentication sources such as LDAP and Active Directory (AD). The Local Only option is selected by default.

Permissions: On the permissions tab, we define if the user is an Admin or User. When we are creating a standard user, there are specific permissions that can be granted.

API Access: We can enable API access for an account by setting this option to Yes. When creating new users the default selected option is No.

Click the Create User button once we have finished populating the fields.

How to create an alert ?

In Nagios Log Server, in the left pane under Alerting click Alerts, then click the New Alert button.

The Create an Alert popup is displayed. The last option Alert Method will show additional options based on the method chosen. All the other options are common to any alert method chosen.

Let us see how to create an alert for each alert method type.

i. Query

ii. Real-Time

iii. Host Freshness

Creating An Alert of the Query method type

To set up an alert with the Query method type, we need to fill in the details in the following format.

i. Alert Name – The descriptive name we want to give this alert.

ii. Type – Select Query

iii. Query – The predefined query we want this alert to be based on. This example is using the Failed SSH Logins query that is included with Nagios Log Server.

iv. Check Interval – This is how often we would like this alert to be checked.

v. Lookback Period – How far in the log data to look back when the query is checked.

vi. Thresholds – This is what defines the severity of the alert. When the query is executed (for the defined lookback period), the number of events returned by the query is the value that the thresholds are tested against. The left field is the warning threshold, the right field is the critical threshold.

There is an additional common option that is not shown until an Alert Method is chosen. “Only alert when Warning or Critical threshold is met” is an important option and our selection depends on our requirements.

When this option is enabled, it:

i. Alerts are only applied to our Alert Method when the warning or critical threshold is met.

ii. We would only receive an alert when there is a problem.

iii. When the problem is no longer occurring, we will not be notified.

When this option is disabled, it:

i. Alerts are applied to our Alert Method regardless if the threshold levels are met.

ii. We will receive an alert every time the alert is run (check interval).

iii. This can be noisy when using email alerts.

iv. If using NRDP, the status in Nagios [XI/Core] will be updated every time the alert is run.

Creating An Alert of the Real-Time method type

To set up an alert with the Real-Time method type, we need to fill in the details in the following format.

i. Alert Name – The descriptive name we want to give this alert.

ii. Type – Select Real-Time

iii. Criteria – This is where we define what fields will trigger this alert. We should be as specific as possible to ensure we do not receive excessive alerts. For example, if the “and” operator is selected, then both the criteria need to match.

Operators

When we select the operator, another field is automatically added.

Each field has a comparison operator that is used to determine if the field is triggered. Suppose the first field uses a string comparison == to match the program name. The second field uses a regular expression =~ to find the phrase Failed password, this has been enclosed in forwarding slashes.

String comparison can be performed with the following operators:

i. == Equals

ii. != Not Equals

iii. =~ Regular Expression Match

iv. !~ Not Regular Expression Match

Operators used for Numeric comparison include:

i. == Equals

ii. != Not Equals

iii. > Greater Than

iv. >= Greater Than Or Equals To

v. < Less Than

vi. <= Less Than Or Equals To

The operators available between fields are as follows:

i. and

ii. or

Rate Limit exists to combat e-mail spam. Alert at most once every n seconds per instance. For example, for a 3-node cluster with a rate limit of 5, we would get a maximum of 3 alerts per 5 seconds.

To have the alert become active immediately, we need to select Save & Apply Configuration. This will restart the Logstash service which can take several minutes to restart. If we are creating multiple alerts at a

time, it is recommended to uncheck this, and then when we have created all of our alerts, navigate to the Configure menu to Apply Configuration.

Creating An Alert of the Host Freshness method type

To set up an alert with the Host Freshness method type, we need to fill in the details in the following format.

i. Alert Name – The descriptive name we want to give this alert.

ii. Type – Select Host Freshness

iii. Hosts – Define which hosts to check using CIDR notation. Multiple subnets can be specified using commas, only IPv4 is supported at this time. We can specify individual hosts by using the /32 subnet mask, for example, 192.168.130.22/32.

iv. Thresholds – This is what triggers the alert. A common use of the host freshness check is to detect when a host is no longer sending logs to Nagios Log Server. By using 0 for both warning and critical, it triggers a critical condition. Nagios Log Server polls for hosts that have not sent data in 24 hours and populate those hosts in a table. Host Freshness alerts run once per hour to check if the host(s) in their configurations are found in that table.

Alert Methods

The final part of creating an alert is to select the alert method and the relevant options.

Nagios (send using NRDP)

i. NRDP Server – This will be populated with the NRDP server(s) we have already added to Nagios Log Server, select the one we are going to send alerts to.

ii. Hostname – The host in Nagios XI or Nagios Core that this alert is going to target.

iii. Servicename – The service in Nagios XI or Nagios Core that this alert is going to target.``````````````````````````````````````````````````

Click the Create Alert button to create a new alert. It will now be displayed under Alerting > Alerts.

Execute Script

i. Script – Add the absolute file path of the script we want to access on our local Nagios Log Server.

ii. Arguments – Here we will indicate what the script will accept as arguments. There is also a list of context variables that will be replaced by the status of the alert being acted upon, these variables can be used in the Arguments field.

Send SNMP Trap

Trap Receiver – It populates with the SNMP Trap server(s) we have already added to Nagios Log Server. Select the one we are going to send alerts to.

Email Users

i. Select Users – Select all the users that we want this alert to be emailed.

ii. Email Template – Select the template that it uses when it send the email.

Alert Actions

Navigate to Alerting > Alerts to see all the alerts that have been defined. There are several options in the Actions column as follows:

i. Show alert in Dashboard: This will open the query used by this alert in the dashboard including the lookback period defined for the alert.

ii. Run the alert now: This causes the alert query to run immediately.

iii. Deactivate/Activate this alert: This allows us to activate or deactivate the alert.

iv. Edit the alert: Make changes to the existing alert we have defined.

v. Remove: This allows us to remove alerts we no longer require.

Alert Query

When adding a New Alert, we will be presented with a drop-down list of already defined queries. After selecting the desired query and creating the alert, this creates a copy of the query we selected.

If we want to update our alert query, edit the existing alert and then click the Advanced (Manage Query) link.

To update the alert to use the new query, select it from the drop-down list and then click the Load button. Alternatively, we can edit the query in the text area field.

Email Templates

Nagios Log Server allows us to create custom email templates, allowing us to have differently formatted alert emails. Navigate to Alerting > Alert Settings > Email Templates.

Email Template Macros

When we are creating email templates, there are macros we can use to add dynamic data to our emails, for example, %state% is the state of the alert (OK/WARNING/CRITICAL/UNKNOWN). The View Macros button provides a list of macros that we can use in the templates along with an explanation.

To create a new template click the + Add Template button.

We will need to populate the Template Name, Subject, and Message Body fields.

The Load button can populate all the fields based on the System Default or Current Default template.

Click the Add button to create the template.

The Email Templates screen shows the newly created template in the list.

The Actions column allows us to Edit and Remove the templates in the list.

For example, suppose that the Default Email Template is currently the System Default. We can change this by clicking the Change link and selecting the preferred template. This setting applies to all alerts that have System Default selected.

We can also modify the actual System Default template by clicking the System Default link above.

Nagios Threshold Values

The Nagios Threshold standards were designed with many different use cases, for example, negative numbers are valid values. However in the case of Nagios Log Server, when it executes an alert query (for the defined loopback period), the number of events that the query returns is the value that the thresholds are tested against. With this in mind, the alert value will always be 0 or greater (no negative numbers involved).

Nagios Passive Services For NRDP

Passive checks are the NRDP alerts that Nagios XI or Nagios Core receives. This means that Nagios XI or Core will need to be configured with services for these passive checks. Otherwise, it will ignore the alerts. Nagios XI has built-in functionality to create services for check results it has received.

In Nagios Core, we will need to create the service definition in our configuration files for these check results.

[Need any further assistance in alerting on Log Events with Nagios Log Server? – We're available 24*7. ]