Install the Splunk App for Infrastructure in a distributed deployment

You can deploy the Splunk App for Infrastructure in your distributed deployment of Splunk Enterprise. To do so, you need to complete these steps:

• Install SAI on the search head tier.
• Install the Splunk Add-on for Infrastructure on the indexer tier.
• Enable the HTTP Event Collector (HEC) on the indexer tier.
• If you want to collect AWS data, deploy or configure a heavy forwarder to handle AWS data collection.

If you want to integrate VMware vCenter Servers in SAI, you have to install additional components. For more information, see About VMware vSphere integrations in SAI in the Administer Splunk App for Infrastructure Manual.

After you complete these steps, start collecting data from systems to monitor in the app. You can run the easy install script, or set up data collection manually. For more information, see the Administer Splunk App for Infrastructure guide.

What the distributed deployment looks like

This diagram describes a distributed environment that is ingesting data from a Windows system, a Mac system, a Linux system, and a heavy forwarder for AWS data collection. Each system sends S2S traffic from a universal forwarder directly to an indexer cluster and HTTP traffic from collectd to a third-party load balancer. The load balancer forwards traffic to HECs in the indexer cluster.

Where to install the app and other dependencies

The following table describes the required locations for installing the Splunk App for Infrastructure and other dependencies in your distributed deployment environment.

Component	Search heads	Indexers	Heavy forwarder	Description
Splunk App for Infrastructure	Required		Required*	*Only when you deploy a heavy forwarder for AWS data collection.
Splunk Add-on for Infrastructure		Required	Required*	You must install the add-on on each indexer to provide props and transforms for data types. *Only when you deploy a heavy forwarder for AWS data collection or use a heavy forwarder as an intermediary before you send data to an indexer.
Splunk Add-on for Amazon Web Services			Required	You must install the add-on if you are collecting data from AWS. Version 5.0.0 is supported.
HTTP Event Collector		Required*		*If you are collecting metrics from a *nix host, this is required. Collectd, which collects metrics data from *nix hosts, sends data to a HEC.
TCP input		Required*		*If you are collecting *nix and Windows logs and Windows metrics, configure a TCP input. You need to configure a port to receive data from a universal forwarder.

Prerequisite

To complete this task, you must be an administrator familiar with clustered environments on Splunk Enterprise.

Steps

Follow these steps to set up the Splunk App for Infrastructure in a distributed Splunk Enterprise deployment.

1. Install the Splunk App for Infrastructure on search heads

Install the Splunk App for Infrastructure on every search head in the cluster. For more details about this task, see Deploy a configuration bundle in the Splunk Enterprise Distributed Search manual.
Prerequisite
First, check that you satisfy the system requirements for the Splunk App for Infrastructure. For more, see System requirements for Splunk App for Infrastructure.

1. Download the Splunk App for Infrastructure from Splunkbase.
2. On the machine that runs the search head cluster's deployer, copy the Splunk_App_Infrastructure directory to the $SPLUNK_HOME/etc/shcluster/apps directory.
3. Push the Splunk App for Infrastructure to every search head in the cluster:

   $SPLUNK_HOME/bin/splunk apply shcluster-bundle     -target <any_cluster_member_mgmt_url:mgmt_port> -auth <username:passwd>
   

2. Install the Splunk Add-on for Infrastructure on indexers

Install the Splunk Add-on for Infrastructure on the indexers. When you install the add-on, it creates the em_metrics and infra_alerts indexes, and handles props and transforms for all data types. For more information about the source types and components that the add-on configures, see Source types and components for the Splunk Add-on for Infrastructure in the Use Splunk Add-on for Infrastructure manual.

For more information about installing the add-on across an indexer cluster, see Update common peer configurations and apps in the Managing Indexers and Clusters of Indexers guide.

1. Download the Splunk Add-on for Infrastructure from Splunkbase.
2. On the machine that runs the indexer cluster manager node, copy the Splunk_TA_Infrastructure directory to the $SPLUNK_HOME/etc/master-apps directory.
3. Validate the bundle and check whether a restart is necessary:

   splunk validate cluster-bundle --check-restart
   

4. Apply the bundle to the indexer cluster:

   splunk apply cluster-bundle
   

5. To see how the cluster bundle push is going, run this command:

   splunk show cluster-bundle-status
   

3. Configure inputs.conf for the indexing tier

Enable receiving on the TCP port for logs and perform metrics for Windows data collection for every indexer in the cluster. To do this, open a receiving port for the indexing tier. For more information about opening a receiving port, see inputs.conf in the Splunk Enterprise Admin Manual.

If you are collecting metrics data from a *nix host, also configure an HEC token. When you configure an HEC token, set the source type to em_metrics, and specify the metrics index you want to use. By default, the metrics index is em_metrics. For more information about configuring an HEC token, see Create an Event Collector token in the Getting Data In guide.

1. On the machine that runs the indexer cluster manager node, go to the $SPLUNK_HOME/etc/system/local directory.
2. open the inputs.conf file with a text editor.
3. Add a [splunktcp] stanza:

   [splunktcp://<port>]
   disabled = 0
   

   where <port> is the port that you want to use to receive data from your host machines. The recommended value is 9997.

   For more information about configuring inputs.conf, see inputs.conf in the Admin Manual.

4. If you are collecting metrics data from a *nix host, add an HEC token stanza:

   [http://<token_name>]
   disabled = 0
   index = em_metrics
   indexes = em_metrics
   sourcetype = em_metrics
   token = <string>
   

   where <token_name> is the name of the token and <string> is a unique identifier for the token value.
5. If you have not already enabled global HEC settings, enable HEC now in an http stanza:

   [http]
   disabled = 0
   

6. Copy the inputs.conf file to the $SPLUNK_HOME/etc/master-apps/_cluster/local directory.

4. Push the indexer cluster master node's configuration bundle to the indexer cluster

Push the Splunk_TA_Infrastructure directory and inputs.conf file to every indexer in the indexer cluster. For more information, see Update common peer configurations and apps.

On the machine that is running the indexer cluster master node, apply the configuration bundle to every indexer in the cluster:

$SPLUNK_HOME/bin/splunk apply cluster-bundle

5. (Optional) Configure a heavy forwarder to collect AWS data

If you haven't already deployed a heavy forwarder that can handle receiving AWS data, first deploy a heavy forwarder. For more information, see Deploy a heavy forwarder in the Splunk Enterprise Forwarding Data guide.

Install these apps and add-ons on the heavy forwarder:

• Splunk App for Infrastructure
• Splunk Add-on for Infrastructure
• Splunk Add-on for AWS

For information about installing apps and add-ons, see Where to get more apps and add-ons in the Splunk Enterprise Admin Manual.

6. Configure data collection

Configure data collection for the Splunk App for Infrastructure using the easy install script under the Add Data tab. You can collect data from Linux, Mac OS X, and Windows hosts, Kubernetes and OpenShift clusters. The easy install script in the Add Data tab cannot set up data forwarding to multiple indexers or a load balancer. If you are sending data to multiple indexers or a load balancer, manually configure data collection.

There are two ways to collect *nix metric and log data in the Splunk App for Infrastructure (SAI). You can use collectd with either the easy install script or manually set up collectd with a universal forwarder. Or, you can collect *nix data with the Splunk Add-on for Unix and Linux installed on the Splunk universal forwarder.

In the Splunk App for Infrastructure (SAI), you can collect VMware Vsphere data with the Splunk Add-on for VMware Metrics.

For information about configuring data collection, see How to add data to Splunk App for Infrastructure in the Administer Splunk App for Infrastructure manual.

For more information on package source, install commands, and install locations for collectd and the collectd plug-in according to the supported operating system, see collectd package sources, install commands, and locations in the Administer Splunk App for Infrastructure manual.