Install the Splunk Add-on for Amazon Kinesis Firehose on to your Splunk platform deployment

Install the Splunk Add-on for Amazon Kinesis Firehose on to your Splunk platform deployment.

Prepare the Splunk Add-on package for installation

Before you deploy the Splunk Add-on, modify the add-on package:

• Remove the eventgen.conf files.
• Remove all files in the samples folder.
• Remove the inputs.conf file.
• Remove the inputs.conf.spec file.

Install on search heads

1. From the Splunk Web home screen, click the gear icon next to Apps.
   
2. Click Install app from file.
3. Locate the downloaded file and click Upload.
4. Restart Splunk Enterprise if prompted.
5. From the Splunk Web home screen, click the gear icon next to Apps.
6. Find the add-on and click Edit properties.
7. Change Visible to Yes.
8. Repeat the above steps for each search head in your Splunk platform deployment.

Install an add-on on clustered indexers in a distributed Splunk Enterprise deployment

Use the master node to deploy add-ons to the peer nodes. Do not use a deployment server or any third-party deployment tool.

Prepare the configuration bundle

The set of subdirectories in the $SPLUNK_HOME/etc/master-apps directory constitute the configuration bundle.

Prepare the configuration bundle by making the following edits to the files you want to distribute to the peers. Try to combine all updates in a single bundle to reduce the impact on the work of the peer nodes:

1. Inspect the add-on for indexes.conf files. For each index defined in an add-on-specific indexes.conf file, set repFactor=auto so that the index is replicated across all peers.
2. Place the add-on in the $SPLUNK_HOME/etc/master-apps directory on the master node.

(Optional) Validate the bundle and check restart

Validate the bundle and test the files on a standalone test indexer to confirm that they are working correctly before distributing them to the set of peers. This helps ensure that the bundle applies across all peer nodes without problems. The validation process also provides information that is useful for debugging invalid bundles.

Use Splunk Web to validate the bundle and check restart

1. On Splunk Web for the master node instance, click Settings > Indexer Clustering.
2. Click Edit > Configuration Bundle Actions.
3. Click Validate and Check Restart > Validate and Check Restart.
   A message appears that indicates bundle validation and whether check restart succeeds.
   When bundle validation and check restart succeeds, then the bundle is acceptable for distribution to the peer nodes. Information about the validated bundle appears in Splunk Web, including whether you must restart the peer nodes.
   
   If validation and check restart fails, then the bundle cannot be distributed to the peers. In this case, review the bundle details for information that might help you troubleshoot the issue.

Use the CLI to validate the bundle and check restart

Run splunk validate cluster-bundle: splunk validate cluster-bundle. This command returns a message confirming that bundle validation has started. Under certain failure conditions, the message indicates the cause of failure.

To validate the bundle and check whether you must restart Splunk, include the --check-restart parameter:
splunk validate cluster-bundle --check-restart This version of the command first validates the bundle, and if validation succeeds, the command checks whether to restart the peer.

To view the status of bundle validation, run the splunk show cluster-bundle-status command. This command shows validation success or failure. If validation fails, the command provides information about the cause of failure and whether you should restart the peer.

The following example shows the output from the splunk show cluster-bundle-status command after a successful validation:

master
	 cluster_status=None 
	 active_bundle
		checksum=576F6BBB187EA6BC99CE0615B1DC151F 
		timestamp=1495569737 (in localtime=Tue May 23 13:02:17 2017) 
	 latest_bundle
		checksum=576F6BBB187EA6BC99CE0615B1DC151F 
		timestamp=1495569737 (in localtime=Tue May 23 13:02:17 2017) 
	 last_validated_bundle
		checksum=1E0C4F0A7363611774E1E65C8B3932CF 
		last_validation_succeeded=1 
		timestamp=1495574646 (in localtime=Tue May 23 14:24:06 2017)
         last_check_restart_bundle
                 checksum=1E0C4F0A7363611774E1E65C8B3932CF 
                 last_check_restart_result=restart required 
                 timestamp=1495574646 (in localtime=Tue May 23 14:24:06 2017) 

Peer 1	 1D00A8C2-026B-4CAF-90D6-5D5D39445569	 default 
	 active_bundle=576F6BBB187EA6BC99CE0615B1DC151F 
	 latest_bundle=576F6BBB187EA6BC99CE0615B1DC151F 
	 last_validated_bundle=1E0C4F0A7363611774E1E65C8B3932CF 
	 last_bundle_validation_status=success
         last_bundle_checked_for_restart=1E0C4F0A7363611774E1E65C8B3932CF 
         last_check_restart_result=restart required
	 restart_required_apply_bundle=0 
	 status=Up 
...

Where the settings are:

Notification field name	Description
last_validated_bundle	Identifies the newly validated bundle.
last_validation_succeeded=1	Indicates that validation succeeded.
last_check_restart_result=restart required	On the master, last_check_restart_result=restart required indicates that a restart is required on at least one of the cluster peers.
last_check_restart_result=restart required	On the peers, last_check_restart_result=restart required indicates that you must restart that peer.

Apply the bundle to the peers

To apply the configuration bundle to the peers, you can use Splunk Web or the CLI. You cannot initiate a configuration bundle push if a bundle push is currently in progress.

Use Splunk Web to apply the bundle to the peer nodes

To apply the configuration bundle to the peer nodes:

1. On the master node, in Splunk Web, click Settings > Indexer clustering.
   
2. Click Edit > Configuration Bundle Actions.
   The configuration bundle actions dashboard opens, and shows information on the last successful bundle push.
   
3. Click Push.
   A pop-up window warns you that the distribution might initiate a restart of all peer nodes.
4. Click Push Changes.
   The screen provides information on the distribution progress and whether distribution is successful.
   • In the case of successful distribution, once each peer successfully validates the bundle, the master coordinates a rolling restart of all the peer nodes as needed.
   • If distribution fails, the master indicates which peers could not receive the distribution so that you can resolve those peer issues. If any peer fails to accept the distribution, none of the peers will apply the bundle.
     

   When the push is successful, the peers use their new set of configurations, now located in their local $SPLUNK_HOME/etc/slave-apps. Leave the files in this location.

Use the CLI to apply the bundle to the peer nodes

1. To apply the configuration bundle to the peers, run the following CLI command on the master:
   splunk apply cluster-bundle
   The warning message appears:

   Caution: Under some circumstances, this command will initiate a rolling restart 
   of all peers. This depends on the contents of the configuration bundle. For 
   details, refer to the documentation. Do you wish to continue? [y/n]:
   

2. To proceed, type y.
   
   • The master distributes the new configuration bundle to the peers, which then individually validate the bundle. After all peers successfully validate the bundle, the master coordinates a rolling restart of all the peer nodes, if necessary. The peers use their new set of configurations, located in their local $SPLUNK_HOME/etc/slave-apps. Leave the files in this location.
   • If any peer is unable to validate the bundle, it sends a message to the master, and the master displays the error on its dashboard in Splunk Web. You must fix any problems noted by the master and rerun splunk apply cluster-bundle.

View the status of the bundle push

View the status of the bundle push using Splunk Web or the CLI.

Use Splunk Web to view the status of the bundle push

Once an app is distributed to the peers, launch and manage the app on each peer using Splunk Web. The apply cluster-bundle command takes an optional flag, --skip-validation, for use in cases where a problem exists in the validation process. Use this flag only under the direction of Splunk Support and after making sure that the bundle is valid. Do not use this flag to circumvent the validation process.

You can also validate the bundle without applying it. This is useful for debugging some validation issues.

Use the CLI to view the status of the bundle push

To see how the cluster bundle push is proceeding, run the following command from the master node:

splunk show cluster-bundle-status

This command tells you whether the bundle validation succeeded or failed. It also indicates the restart status of each peer.

1. Download the add-on from Splunkbase.
2. Extract the add-on.
3. Place the resulting Splunk_TA_<add-on_name> folder in the $SPLUNK_HOME/etc/apps directory on your heavy forwarder.
4. Restart the heavy forwarder using the command splunk restart.