Configure Linux Data Collection for Splunk App for Infrastructure

Admin privileges are required to configure data collection.

To monitor your entities, install a data collection agent installation script (a collectd daemon and universal forwarder) on the host machine from which you are collecting data. Data, such as metrics, logs, and events, are forwarded to Splunk App for Infrastructure for performance monitoring and investigation of your infrastructure.

Video demonstration

For a video demonstration for how to configure log and metric collection, see Video: Configuring log and metrics collection for Linux.

Prerequisites

The agent and installation script require the following. These requirements are also defined in the Agent Installation Script system requirements.

Item	Requires
Linux machine	Supported operating systems include: RHEL 6.2.6 RHEL 7.3.10 CentOS 6 and CentOS 7 Debian 8 Ubuntu 16
Installation script	Script must be run by a user with root privileges. Entity should have these dependencies available: wget apt-get (Debian and Ubuntu) yum (Redhat and Centos) Internet access

Steps

Step 1: Specify configuration options

Select and/or customize your data collection options for collecting metrics and logs from your host.

1. In the Splunk App for Infrastructure user interface, click the Add Data tab.
2. In the left panel click Linux.
3. Customize the Data to be collected. Click the Customize link.
   • When you select or customize the data to be collected, this also customizes the agent installation script in Step 2 that you run on your host machine.
4. Select the metrics and log sources for which you want to collect data.
   • The metric cpu is selected by default.
   • If selecting cpu > Collect data for each CPU, metrics are stored for each cpu individually, which enables you to use the Split-by feature in the Analysis Workspace.
   • If selecting cpu > Collect sum over all CPUs, only aggregate metrics are stored.
5. Click Save.
6. Add Dimensions for easier troubleshooting, analysis, and filtering hosts.
   • Dimensions are key/value pairs that provide metadata about the metric (describes the measurement) used for searching and filtering relevant datasets (distinct time series) during an investigation.
   • Use the format of dimension:value, such as env:prod.
7. Enter the Monitoring machine hostname or IP address and port number of the machine that has Splunk App for Infrastructure installed (the machine that you are sending data to). For example, my.instance.domain.name.
   • Specify the HEC port (HTTP Event Collector Port) of the machine you want to send metric data to. For example, 8088.
   • Specify the Receiver port of the machine you want to send log data to. For example, 9997.
   • Specify the HEC token of the machine you want to send data to. To create an HEC token, see Create an Event Collector token.

Step 2: Copy and paste the following into the command line of your entity

Deploy the Agent Installation Script on your host to collect metrics and logs.

• This script must be run by a user with root access.
• This script is customized by selecting metrics and log sources from the Customize link/dialog when specifying configuration options.
• Entity should have these dependencies available: wget, apt-get (Debian and Ubuntu), yum (Redhat and Centos), and Internet access.
• See Agent Installation Script Actions below for information about the actions this script performs.

1. Open a terminal window on the host machine (the machine that has your data).
2. Secure shell (SSH) into your host machine. You need root access to run the script.
3. Paste the Agent Installation Script (includes the collectd daemon and universal forwarder) into the terminal window. If you want to customize this script, see the Customize link/dialog.
4. Run the script. If you are running the script for the first time, see the following note about creating administrator credentials.
   • When the Agent Installation Script is run for the first time, you might receive this message: "This appears to be your first time running this version of Splunk. IMPORTANT: Because an admin password was not provided, the admin user
 will not be created. You will have to set up an admin username/password
 using user-seed.conf." If you receive this message, it means the Agent Installation Script has installed the universal forwarder without creating the admin user. To enable the admin user on the universal forwarder in the event you want to run splunkforwarder CLI commands, you must manually create the administrator credentials. To resolve this issue, see Create Administrator Credentials Manually later in this topic for instruction on creating the credentials.

Step 3: Once the script finishes running, verify your data connection

Verify your data connection to start monitoring your infrastructure.

It can take up to about five (5) minutes for your entities to display in the user interface.

1. In the Splunk App for Infrastructure user interface, return to your web browser and the Add Data view.
2. When the script finishes running, the user interface indicates your entity is connected and data is available to view.
   • If no new entities are connected after a few minutes, click Refresh.
   • When new entities are connected, click New host found to view your entity.

Summary

When you have set up the data collection agent on your host machine, and validate new entities are connected, you can start monitoring your infrastructure. Go to the Investigate page to monitor your entities in the Infrastructure Overview or List View. You can group your entities to monitor them more easily, and drilldown to the Overview Dashboard (entities only) or Analysis Workspace (entities and groups) to further analyze your infrastructure.

Agent Installation Script Actions

The agent installation script, from Step 2 above, performs the following:

• Checks system for dependencies
• Installs collectd
• Adds write_splunk plugin to collectd package
• Configures collectd (link to collectd configuraiton file example)
• Starts collectd
• Downloads splunk Universal Forwarder
• Configures splunk Universal Forwarder
• Starts Splunk Universal Forwarder. If no log sources are selected, the Universal Forwarder will not be installed.

Create Administrator Credentials Manually

As explained in Step 2 in this topic, in order to log in as an admin user to run splunkforwarder CLI commands, you must manually create the universal forwarder administrator credentials. Follow the steps below and restart Splunk App for Infrastructure.

1. Stop Splunk App for Infrastructure:

   ./splunk stop
   

2. With a text editor, create $SPLUNK_HOME/etc/system/local/user-seed.conf, substituting $SPLUNK_HOME for where you installed the software.
3. Within the file, add the following lines, substituting a password for your new password:

   [user_info]
   USERNAME = admin
   PASSWORD = <your new password>
   

4. Save the file and close it.
5. Restart Splunk.