Install the Splunk Add-on for Nagios Core on a distributed Splunk Enterprise deployment

1. Get the Splunk Add-on for Nagios Core by downloading it from https://splunkbase.splunk.com/app/2703 or browsing to it using the app browser within Splunk Web.
2. Determine where and how to install this add-on in your deployment, using the tables on this page.
3. Perform any prerequisite steps before installing, if required and specified in the tables below.
4. Complete your installation.

If you need step-by-step instructions on how to install an add-on in your specific deployment environment, see the installation walkthroughs section at the bottom of this page for links to installation instructions specific to a single-instance deployment, distributed deployment, or Splunk Cloud.

Prepare the Splunk Add-on for Nagios Core package for installation in a distributed Splunk Enterprise deployment with indexer clustering

Before deploying the Splunk Add-on for Nagios Core in a distributed Splunk Enterprise deployment with indexer clustering, make the following changes to the Nagios Core add-on package:

• Remove the eventgen.conf files.
• Remove all files in the samples folder.

Install the Splunk Add-on for Nagios Core on search heads in a distributed Splunk Enterprise deployment

1. From the Splunk Web home screen, click the gear icon next to Apps.
   
2. Click Install app from file.
3. Locate the downloaded file and click Upload.
4. If Splunk Enterprise prompts you to restart, do so.
5. From the Splunk Web home screen, click the gear icon next to Apps.
6. Find the add-on and click Edit properties.
7. Change Visible to No.

Install an add-on on clustered indexers in a distributed Splunk Enterprise deployment

Use the master node to deploy add-ons to the peer nodes. Do not use a deployment server or any third-party deployment tool.

Prepare the configuration bundle

The set of subdirectories in the $SPLUNK_HOME/etc/master-apps directory constitute the configuration bundle.

Prepare the configuration bundle by making the following edits to the files you want to distribute to the peers. Try to combine all updates in a single bundle to reduce the impact on the work of the peer nodes:

1. Inspect the add-on for indexes.conf files. For each index defined in an add-on-specific indexes.conf file, set repFactor=auto so that the index is replicated across all peers.
2. Place the add-on in the $SPLUNK_HOME/etc/master-apps directory on the master node.

(Optional) Validate the bundle and check restart

Validate the bundle and test the files on a standalone test indexer to confirm that they are working correctly before distributing them to the set of peers. This helps ensure that the bundle applies across all peer nodes without problems. The validation process also provides information that is useful for debugging invalid bundles.

Use Splunk Web to validate the bundle and check restart

1. On Splunk Web for the master node instance, click Settings > Indexer Clustering.
2. Click Edit > Configuration Bundle Actions.
3. Click Validate and Check Restart > Validate and Check Restart.
   A message appears that indicates bundle validation and whether check restart succeeds.
   When bundle validation and check restart succeeds, then the bundle is acceptable for distribution to the peer nodes. Information about the validated bundle appears in Splunk Web, including whether you must restart the peer nodes.
   
   If validation and check restart fails, then the bundle cannot be distributed to the peers. In this case, review the bundle details for information that might help you troubleshoot the issue.

Use the CLI to validate the bundle and check restart

Run splunk validate cluster-bundle: splunk validate cluster-bundle. This command returns a message confirming that bundle validation has started. Under certain failure conditions, the message indicates the cause of failure.

To validate the bundle and check whether you must restart Splunk, include the --check-restart parameter:
splunk validate cluster-bundle --check-restart This version of the command first validates the bundle, and if validation succeeds, the command checks whether to restart the peer.

To view the status of bundle validation, run the splunk show cluster-bundle-status command. This command shows validation success or failure. If validation fails, the command provides information about the cause of failure and whether you should restart the peer.

The following example shows the output from the splunk show cluster-bundle-status command after a successful validation:

master
	 cluster_status=None 
	 active_bundle
		checksum=576F6BBB187EA6BC99CE0615B1DC151F 
		timestamp=1495569737 (in localtime=Tue May 23 13:02:17 2017) 
	 latest_bundle
		checksum=576F6BBB187EA6BC99CE0615B1DC151F 
		timestamp=1495569737 (in localtime=Tue May 23 13:02:17 2017) 
	 last_validated_bundle
		checksum=1E0C4F0A7363611774E1E65C8B3932CF 
		last_validation_succeeded=1 
		timestamp=1495574646 (in localtime=Tue May 23 14:24:06 2017)
         last_check_restart_bundle
                 checksum=1E0C4F0A7363611774E1E65C8B3932CF 
                 last_check_restart_result=restart required 
                 timestamp=1495574646 (in localtime=Tue May 23 14:24:06 2017) 

Peer 1	 1D00A8C2-026B-4CAF-90D6-5D5D39445569	 default 
	 active_bundle=576F6BBB187EA6BC99CE0615B1DC151F 
	 latest_bundle=576F6BBB187EA6BC99CE0615B1DC151F 
	 last_validated_bundle=1E0C4F0A7363611774E1E65C8B3932CF 
	 last_bundle_validation_status=success
         last_bundle_checked_for_restart=1E0C4F0A7363611774E1E65C8B3932CF 
         last_check_restart_result=restart required
	 restart_required_apply_bundle=0 
	 status=Up 
...

Where the settings are:

Notification field name	Description
last_validated_bundle	Identifies the newly validated bundle.
last_validation_succeeded=1	Indicates that validation succeeded.
last_check_restart_result=restart required	On the master, last_check_restart_result=restart required indicates that a restart is required on at least one of the cluster peers.
last_check_restart_result=restart required	On the peers, last_check_restart_result=restart required indicates that you must restart that peer.

Apply the bundle to the peers

To apply the configuration bundle to the peers, you can use Splunk Web or the CLI. You cannot initiate a configuration bundle push if a bundle push is currently in progress.

Use Splunk Web to apply the bundle to the peer nodes

To apply the configuration bundle to the peer nodes:

1. On the master node, in Splunk Web, click Settings > Indexer clustering.
   
2. Click Edit > Configuration Bundle Actions.
   The configuration bundle actions dashboard opens, and shows information on the last successful bundle push.
   
3. Click Push.
   A pop-up window warns you that the distribution might initiate a restart of all peer nodes.
4. Click Push Changes.
   The screen provides information on the distribution progress and whether distribution is successful.
   • In the case of successful distribution, once each peer successfully validates the bundle, the master coordinates a rolling restart of all the peer nodes as needed.
   • If distribution fails, the master indicates which peers could not receive the distribution so that you can resolve those peer issues. If any peer fails to accept the distribution, none of the peers will apply the bundle.
     

   When the push is successful, the peers use their new set of configurations, now located in their local $SPLUNK_HOME/etc/slave-apps. Leave the files in this location.

Use the CLI to apply the bundle to the peer nodes

1. To apply the configuration bundle to the peers, run the following CLI command on the master:
   splunk apply cluster-bundle
   The warning message appears:

   Caution: Under some circumstances, this command will initiate a rolling restart 
   of all peers. This depends on the contents of the configuration bundle. For 
   details, refer to the documentation. Do you wish to continue? [y/n]:
   

2. To proceed, type y.
   
   • The master distributes the new configuration bundle to the peers, which then individually validate the bundle. After all peers successfully validate the bundle, the master coordinates a rolling restart of all the peer nodes, if necessary. The peers use their new set of configurations, located in their local $SPLUNK_HOME/etc/slave-apps. Leave the files in this location.
   • If any peer is unable to validate the bundle, it sends a message to the master, and the master displays the error on its dashboard in Splunk Web. You must fix any problems noted by the master and rerun splunk apply cluster-bundle.

View the status of the bundle push

View the status of the bundle push using Splunk Web or the CLI.

Use Splunk Web to view the status of the bundle push

Once an app is distributed to the peers, launch and manage the app on each peer using Splunk Web. The apply cluster-bundle command takes an optional flag, --skip-validation, for use in cases where a problem exists in the validation process. Use this flag only under the direction of Splunk Support and after making sure that the bundle is valid. Do not use this flag to circumvent the validation process.

You can also validate the bundle without applying it. This is useful for debugging some validation issues.

Use the CLI to view the status of the bundle push

To see how the cluster bundle push is proceeding, run the following command from the master node:

splunk show cluster-bundle-status

This command tells you whether the bundle validation succeeded or failed. It also indicates the restart status of each peer.

1. Download the add-on from Splunkbase.
2. Extract the add-on.
3. Place the resulting Splunk_TA_<add-on_name> folder in the $SPLUNK_HOME/etc/apps directory on your heavy forwarder.
4. Restart the heavy forwarder using the command splunk restart.

Distributed deployments

Use the tables below to determine where and how to install this add-on in a distributed deployment of Splunk Enterprise or any deployment for which you are using forwarders to get your data in. Depending on your environment, your preferences, and the requirements of the add-on, you may need to install the add-on in multiple places.

Where to install this add-on

Unless otherwise noted, all supported add-ons can be safely installed to all tiers of a distributed Splunk platform deployment. See Where to install Splunk add-ons in Splunk Add-ons for more information.

This table provides a reference for installing this specific add-on to a distributed deployment of the Splunk platform.

Splunk platform instance type	Supported	Required	Actions required / Comments
Search Heads	Yes	Yes	Install this add-on to all search heads where Nagios Core knowledge management is required.
Indexers	Yes	Conditional	Not required if you use heavy forwarders to collect all data. Required if you use universal or light forwarders to collect log data.
Heavy Forwarders	Yes	See comments	To collect NDOUtils data, you must use Splunk DB Connect on a search head or heavy forwarder. You can collect log data using a universal or light forwarder installed directly on the machines running Nagios Core.
Universal Forwarders	Yes	See comments	Supported for collecting log data only. The forwarder collecting your log data must be installed on the same machine as your Nagios Core instance.
Light Forwarders	Yes	See comments	Supported for collecting log data only. The forwarder collecting your log data must be installed on the same machine as your Nagios Core instance.

Distributed deployment feature compatibility

This table describes the compatibility of this add-on with Splunk distributed deployment features.

Distributed deployment feature	Supported	Actions required
Search Head Clusters	Yes	You can install this add-on on a search head cluster for all search-time functionality, but configure inputs on forwarders to avoid duplicate data collection. Before you install this add-on to a cluster, remove the eventgen.conf file and all files in the samples folder.
Indexer Clusters	Yes	Before you install this add-on to a cluster, remove the eventgen.conf file and all files in the samples folder.
Deployment Server	No	Supported for deploying unconfigured add-ons only. Using a deployment server to deploy the configured add-on to multiple forwarders acting as data collectors causes duplication of data. The add-on uses the credential vault to secure your credentials, and this credential management solution is incompatible with the deployment server.

Installation walkthroughs

The Splunk Add-Ons manual includes an Installing add-ons guide that helps you successfully install any Splunk-supported add-on to your Splunk platform.

For a walkthrough of the installation procedure, follow the link that matches your deployment scenario:

• Single-instance Splunk Enterprise
• Distributed Splunk Enterprise