Install the Splunk App for Infrastructure in a distributed deployment

This topic is for advanced administrators who are familiar with clustered environments on Splunk Enterprise.

If you plan to install the Splunk App for Infrastructure in a distributed Splunk Enterprise environment, you must install the app in a different way than when you install on a single host. This topic guides you through setting up where to install the Splunk App for Infrastructure and its components, including the Splunk Add-on for Infrastructure and optional Splunk Add-on for AWS.

Example distributed deployment

The following is an example of the Splunk App for Infrastructure and components in a Splunk Enterprise distributed deployment.

Where to install the App, Add-ons, and other dependencies

The following table shows the required and recommended locations for installing the Splunk App for Infrastructure and components in your distributed deployment environment.

Recommended Splunk App for Infrastructure component installation locations

Splunk Component	Search Heads	Indexers	Heavy Forwarder*	*If using a heavy forwarder as an intermediary collection tier, the recommended approach is:
Splunk App for Infrastructure	Required		Required	splunk_app_infrastructure is used to set up data collection via the user interface on the heavy forwarder.
Splunk Add-on for Infrastructure		Required	Required	Splunk_TA_infrastructure must be installed on the heavy forwarder to provide props and transforms for all data types.
Splunk Add-on for AWS			Optional	Splunk_TA_AWS is only required if collecting data from AWS CloudWatch. The Splunk Add-on for AWS 4.5 is supported. The Splunk Add-on for AWS 4.6 is not supported.

Other dependencies

Item	Required	Comments
HTTP input and token for collectd metric collection for Linux and Mac OSX data collection	Yes	See Create the HTTP Event Collector (HEC) token.
TCP input enabled for receiving logs and perfmon metrics for Windows data collection	Yes	See Configure the receiving port.

Steps

Follow these steps to get started with the Splunk App for Infrastructure in a distributed Splunk Enterprise deployment.

You can also send data directly to indexers or search heads. Using the steps listed in this section, make sure the HEC token and inputs are configured on the instance that will receive the data. If this is a search head, you need to configure the search head to send the data to your indexer tier using the forwarding method.

1. Install the Splunk App for Infrastructure on search heads

Install the Splunk App for Infrastructure (app) on the search heads.

1. Using the command line, copy the Splunk App for Infrastructure package to the deployer's search head cluster apps directory.

   /opt/splunk/etc/shcluster/apps
   

2. Push the Splunk App for Infrastructure to all search heads.

   /opt/splunk/bin/splunk apply shcluster-bundle     -target <any_cluster_member_mgmt_url:mgmt_port> -auth <username:passwd>
   

3. Confirm you want to continue.
   • A message displays indicating the package is successfully pushed to all cluster members. Confirm you want to continue.
   • When the app is pushed to the search head cluster, the app displays in the Splunk Enterprise user interface (Splunk Web) for all search heads in the cluster.
   • For more details about this task, see Deploy a configuration bundle.

2. Install the Splunk Add-on for Infrastructure on indexers

Install the Splunk Add-on for Infrastructure on the indexers. When you install the add-on, it creates the em_metrics and infra_alerts indexes. For more information about the source types and components that the add-on configures, see Source types and components for the Splunk Add-on for Infrastructure.

1. Using the command line, copy the Splunk Add-on for Infrastructure to the cluster master.

   /opt/splunk/etc/master-apps
   

2. In Splunk Web, open of the cluster master, and click Settings > Indexer Clustering.
3. Click Edit from the dropdown.
4. Click Configuration Bundle Actions.
5. Click Validate and Check Restart and verify success.
6. Click Push. This pushes the Splunk Add-on for Infrastructure to all the Indexers.

3. Deploy a heavy forwarder

Install Splunk Enterprise and configure it as a heavy forwarder. This instance hosts the data ingestion management components, as well as the HTTP event collector receiver.

1. Install a full Splunk Enterprise instance. See Installation instructions in the Splunk Enterprise Installation Manual.

4. Install the Splunk App for Infrastructure on the heavy forwarder

Follow these steps to install the Splunk App for Infrastructure on the heavy forwarder:

1. Download the Splunk App for Infrastructure from Splunkbase.
2. Log in to the heavy forwarder as an administrator.
3. From the Splunk Web home screen, click the gear icon next to Apps.
4. Click Install app from file.
5. Locate the downloaded file and click Upload.
6. Restart when prompted.

5. Install the Splunk Add-on for Infrastructure on the heavy forwarder

Follow these steps to install the Splunk Add-on for Infrastructure on the heavy forwarder:

1. Download the Splunk Add-on for Infrastructure from Splunkbase.
2. Log in to the heavy forwarder as an administrator.
3. From the Splunk Web home screen, click the gear icon next to Apps.
4. Click Install app from file.
5. Locate the downloaded file and click Upload.
6. Restart when prompted.

6. (Optional) Install the Splunk Add-on for AWS on the heavy forwarder

If collecting AWS CloudWatch data from your AWS instance, follow these steps to install the Splunk Add-on for AWS on the heavy forwarder:

1. Download the Splunk Add-on for AWS from Splunkbase.
2. Log in to the heavy forwarder as an administrator.
3. From the Splunk Web home screen, click the gear icon next to Apps.
4. Click Install app from file.
5. Locate the downloaded file and click Upload.
6. Restart when prompted.

7. Configure the receiving port

Enable receiving on the TCP port for logs and perform metrics for Windows data collection.

1. In Splunk Web, log in to the heavy forwarder as an administrator.
2. Click Settings > Forwarding and receiving.
3. At Configure receiving, click Add new.
4. Specify the TCP port you want the receiver to listen on (the listening port, also known as the receiving port). The recommended port is 9997. For example, if you enter "9997," the receiver listens for connections from forwarders on port 9997. You can specify any unused port. You can use a tool like netstat to determine what ports are available on your system. Make sure the port you select is not in use by splunkweb or splunkd.
5. Click Save. Splunk software starts listening for incoming data on the port you specified.

8. Create the HTTP Event Collector (HEC) token

To complete this step, you should already be familiar with creating an HEC token in Splunk Enterprise. For information about configuring an HEC token, see Create an Event Collector token.

For Linux and Mac OSX data collection, create the HEC input and token for collectd metric collection. For Windows and AWS data collection, skip this step.

1. In Splunk Web, log in as an administrator.
2. Go to Settings > Data inputs, select HTTP Event Collector, and click New Token.
3. For Select Source, do not check Enable indexer acknowledgement.
4. For Input Settings, these are the required settings:
   If the em_metrics source type is not available in the drop-down menu, enter the name of the source type in the filter box to select it.
   
• Source type: em_metrics
• App context: Splunk_TA_Infrastructure
• Index: em_metrics
5. Generate the HEC Token to send data over HEC to the Splunk Enterprise instance.
6. Confirm the token was created and take note of the Token Value.
7. Go to Settings > Data inputs, select HTTP Event Collector, and click Global Settings. Take note of the HTTP Port Number; you will need it later when you start adding data.

9. Configure forwarding to the indexing tier

Configure forwarding to the indexing tier.

1. In Splunk Web, login as an administrator on the instance that will be forwarding data.
2. Click the Settings > Forwarding and receiving.
3. Click Add new at Configure forwarding.
4. Enter the hostname or IP address for the receiving Splunk instance(s), along with the receiving port specified when the receiver was configured. For example, you might enter: receivingserver.com:9997. To implement load-balanced forwarding, you can enter multiple hosts as a comma-separated list.
5. Click Save.

10. Configure data collection

Configure data collection for the Splunk App for Infrastructure using the Add Data window. You can collect data from Linux, Mac OS X, and Windows hosts. if you installed and configured the Splunk Add-on for AWS, you can also collect data from your AWS accounts.

When you configure data collection, view any host from which you're collecting data on the Investigate and Analysis Workspace pages in the app.

For information about configuring data collection, see How to add data to Splunk App for Infrastructure.