Install the Splunk Add-on for AWS

Install the Splunk Add-on for AWS on your distributed Splunk Enterprise deployment. The Splunk Add-on for AWS does not need to be configured, but it contains files that allow your data to be ingested through the Splunk Add-on for Kinesis Firehose.

Prerequisites for adding Amazon Web Services data into a distributed Splunk Enterprise deployment with indexer clustering

For more information about supported data sources, deployment scenarios, and an overview of Guided Data Onboarding, see the Additional resources topic in this manual.

If you want to get Amazon Web Services data into a distributed Splunk Enterprise deployment with indexer clustering, you must have the following prerequisites:

• A distributed Splunk Enterprise deployment with indexer clustering enabled.
• Access to Splunk Web.
• A user role that permits installing apps and add-ons.

Prepare the Splunk Add-on package for installation

Before you deploy the Splunk Add-on, modify the add-on package:

• Remove the eventgen.conf files.
• Remove all files in the samples folder.
• Remove the inputs.conf file.
• Remove the inputs.conf.spec file.

Install an add-on on clustered indexers in a distributed Splunk Enterprise deployment

Use the master node to deploy add-ons to the peer nodes. Do not use a deployment server or any third-party deployment tool.

Prepare the configuration bundle

The set of subdirectories in the $SPLUNK_HOME/etc/master-apps directory constitute the configuration bundle.

Prepare the configuration bundle by making the following edits to the files you want to distribute to the peers. Try to combine all updates in a single bundle to reduce the impact on the work of the peer nodes:

1. Inspect the add-on for indexes.conf files. For each index defined in an add-on-specific indexes.conf file, set repFactor=auto so that the index is replicated across all peers.
2. Place the add-on in the $SPLUNK_HOME/etc/master-apps directory on the master node.

(Optional) Validate the bundle and check restart

Validate the bundle and test the files on a standalone test indexer to confirm that they are working correctly before distributing them to the set of peers. This helps ensure that the bundle applies across all peer nodes without problems. The validation process also provides information that is useful for debugging invalid bundles.

Use Splunk Web to validate the bundle and check restart

1. On Splunk Web for the master node instance, click Settings > Indexer Clustering.
2. Click Edit > Configuration Bundle Actions.
3. Click Validate and Check Restart > Validate and Check Restart.
   A message appears that indicates bundle validation and whether check restart succeeds.
   When bundle validation and check restart succeeds, then the bundle is acceptable for distribution to the peer nodes. Information about the validated bundle appears in Splunk Web, including whether you must restart the peer nodes.
   
   If validation and check restart fails, then the bundle cannot be distributed to the peers. In this case, review the bundle details for information that might help you troubleshoot the issue.

Use the CLI to validate the bundle and check restart

Run splunk validate cluster-bundle: splunk validate cluster-bundle. This command returns a message confirming that bundle validation has started. Under certain failure conditions, the message indicates the cause of failure.

To validate the bundle and check whether you must restart Splunk, include the --check-restart parameter:
splunk validate cluster-bundle --check-restart This version of the command first validates the bundle, and if validation succeeds, the command checks whether to restart the peer.

To view the status of bundle validation, run the splunk show cluster-bundle-status command. This command shows validation success or failure. If validation fails, the command provides information about the cause of failure and whether you should restart the peer.

The following example shows the output from the splunk show cluster-bundle-status command after a successful validation:

master
	 cluster_status=None 
	 active_bundle
		checksum=576F6BBB187EA6BC99CE0615B1DC151F 
		timestamp=1495569737 (in localtime=Tue May 23 13:02:17 2017) 
	 latest_bundle
		checksum=576F6BBB187EA6BC99CE0615B1DC151F 
		timestamp=1495569737 (in localtime=Tue May 23 13:02:17 2017) 
	 last_validated_bundle
		checksum=1E0C4F0A7363611774E1E65C8B3932CF 
		last_validation_succeeded=1 
		timestamp=1495574646 (in localtime=Tue May 23 14:24:06 2017)
         last_check_restart_bundle
                 checksum=1E0C4F0A7363611774E1E65C8B3932CF 
                 last_check_restart_result=restart required 
                 timestamp=1495574646 (in localtime=Tue May 23 14:24:06 2017) 

Peer 1	 1D00A8C2-026B-4CAF-90D6-5D5D39445569	 default 
	 active_bundle=576F6BBB187EA6BC99CE0615B1DC151F 
	 latest_bundle=576F6BBB187EA6BC99CE0615B1DC151F 
	 last_validated_bundle=1E0C4F0A7363611774E1E65C8B3932CF 
	 last_bundle_validation_status=success
         last_bundle_checked_for_restart=1E0C4F0A7363611774E1E65C8B3932CF 
         last_check_restart_result=restart required
	 restart_required_apply_bundle=0 
	 status=Up 
...

Where the settings are:

Notification field name	Description
last_validated_bundle	Identifies the newly validated bundle.
last_validation_succeeded=1	Indicates that validation succeeded.
last_check_restart_result=restart required	On the master, last_check_restart_result=restart required indicates that a restart is required on at least one of the cluster peers.
last_check_restart_result=restart required	On the peers, last_check_restart_result=restart required indicates that you must restart that peer.

Apply the bundle to the peers

To apply the configuration bundle to the peers, you can use Splunk Web or the CLI. You cannot initiate a configuration bundle push if a bundle push is currently in progress.

Use Splunk Web to apply the bundle to the peer nodes

To apply the configuration bundle to the peer nodes:

1. On the master node, in Splunk Web, click Settings > Indexer clustering.
   
2. Click Edit > Configuration Bundle Actions.
   The configuration bundle actions dashboard opens, and shows information on the last successful bundle push.
   
3. Click Push.
   A pop-up window warns you that the distribution might initiate a restart of all peer nodes.
4. Click Push Changes.
   The screen provides information on the distribution progress and whether distribution is successful.
   • In the case of successful distribution, once each peer successfully validates the bundle, the master coordinates a rolling restart of all the peer nodes as needed.
   • If distribution fails, the master indicates which peers could not receive the distribution so that you can resolve those peer issues. If any peer fails to accept the distribution, none of the peers will apply the bundle.
     

   When the push is successful, the peers use their new set of configurations, now located in their local $SPLUNK_HOME/etc/slave-apps. Leave the files in this location.

Use the CLI to apply the bundle to the peer nodes

1. To apply the configuration bundle to the peers, run the following CLI command on the master:
   splunk apply cluster-bundle
   The warning message appears:

   Caution: Under some circumstances, this command will initiate a rolling restart 
   of all peers. This depends on the contents of the configuration bundle. For 
   details, refer to the documentation. Do you wish to continue? [y/n]:
   

2. To proceed, type y.
   
   • The master distributes the new configuration bundle to the peers, which then individually validate the bundle. After all peers successfully validate the bundle, the master coordinates a rolling restart of all the peer nodes, if necessary. The peers use their new set of configurations, located in their local $SPLUNK_HOME/etc/slave-apps. Leave the files in this location.
   • If any peer is unable to validate the bundle, it sends a message to the master, and the master displays the error on its dashboard in Splunk Web. You must fix any problems noted by the master and rerun splunk apply cluster-bundle.

View the status of the bundle push

View the status of the bundle push using Splunk Web or the CLI.

Use Splunk Web to view the status of the bundle push

Once an app is distributed to the peers, launch and manage the app on each peer using Splunk Web. The apply cluster-bundle command takes an optional flag, --skip-validation, for use in cases where a problem exists in the validation process. Use this flag only under the direction of Splunk Support and after making sure that the bundle is valid. Do not use this flag to circumvent the validation process.

You can also validate the bundle without applying it. This is useful for debugging some validation issues.

Use the CLI to view the status of the bundle push

To see how the cluster bundle push is proceeding, run the following command from the master node:

splunk show cluster-bundle-status

This command tells you whether the bundle validation succeeded or failed. It also indicates the restart status of each peer.

To install a heavy forwarder using Linux and connect it to your Splunk platform deployment, perform the following steps:

1. Download and install a full Splunk Enterprise instance.
2. Enable your Splunk Enterprise instance as a heavy forwarder.

Install and configure a heavy forwarder for Linux

Download the Linux version of Splunk Enterprise.

When you install Splunk Enterprise, note the following:

• Some non-GNU versions of tar might not have the -C argument available. In this case, to install in /opt/splunk, you can change directories to /opt or place the tar file in /opt before you run the tar command. This method works for any accessible directory on your host file system.
• Splunk Enterprise does not create the splunk user. To make Splunk Enterprise run as a specific user, you create the user manually before you install.
• Confirm that the disk partition has enough space to hold the uncompressed volume of the data you plan to keep indexed.

To install Splunk Enterprise, follow these steps:

1. Untar the Splunk Enterprise file into an appropriate directory:

   tar xvzf splunk_package_name.tgz
   

   The default installation directory is splunk in the current working directory. To install into /opt/splunk, use the following command:

   tar xvzf splunk_package_name.tgz -C /opt
   

2. Navigate to the directory where you installed Splunk, and start the Splunk software.

   ./splunk start
   




3. A command line window prompts you to create an administrator password. Type the password when prompted. You need this password for your initial Splunk Enterprise login.

   This appears to be your first time running this version of Splunk.
   
   An Admin password must be set before installation proceeds.
   

If you used the --no prompt argument in the command line to start Splunk Enterprise, you are not prompted to create the administrator credentials needed to log into Splunk Enterprise for the first time.

Enable your Splunk Enterprise instance as a heavy forwarder

You can use Splunk Web or the CLI to enable forwarding for a Splunk instance.

Set up heavy forwarding with Splunk Web

Per the previous steps, you should already be logged into Splunk Web as admin on the instance that will be forwarding data.

1. If necessary, log into Splunk Web as admin on the instance that will be forwarding data.
2. Click Settings > Forwarding and receiving.
3. At Configure forwarding, click Add new.
4. Enter the hostname or IP address for the receiving Splunk instance(s), along with the receiving port specified when the receiver was configured. For example, you might enter receivingserver.com:9997.
5. Click Save.
6. Restart Splunk Web.

Configure heavy forwarders to index and forward data

Use a heavy forwarder to receive, parse and forward the data to another indexer.

1. Log into Splunk Web as admin on the instance that will be forwarding data.
2. Click Settings > Forwarding and receiving.
3. Select Forwarding defaults.
4. Select Yes to store and maintain a local copy of the indexed data on the forwarder.

Set up heavy forwarding with the CLI

In the command line, enable forwarding on the Splunk Enterprise instance, then configure forwarding to a specified receiver.

1. From a command or shell prompt, navigate to $SPLUNK_HOME/bin/.
2. Type the following command to enable forwarding:

   splunk enable app SplunkForwarder -auth <username>:<password>
   

3. Restart Splunk Enterprise.

Start forwarding using the CLI

Send data to the receiving indexer that you specify.

1. From a shell or command prompt, go to the $SPLUNK_HOME/bin directory.
2. Specify the receiver with the splunk add forward-server command:

   splunk add forward-server <host>:<port> -auth <username>:<password>
   

3. Restart the forwarder.

1. Download the add-on from Splunkbase.
2. Extract the add-on.
3. Place the resulting Splunk_TA_<add-on_name> folder in the $SPLUNK_HOME/etc/apps directory on your heavy forwarder.
4. Restart the heavy forwarder using the command splunk restart.