Collect Linux/Unix metrics and logs with Splunk App for Infrastructure

To configure data collection, you must log in to an account with permissions to use sudo for root access. Do not log in as the root user.

Use the easy install script to install and configure data collection agents on a host you want to collect metrics and log data from. You can forward metrics and log data to the Splunk App for Infrastructure (SAI) to monitor performance and investigate your infrastructure. If you are running Docker containers without an orchestration tool like Docker Swarm, Kubernetes, or OpenShift, you can use the script to monitor the Docker containers as well.

To manually configure data collection, see Manually configure log collection on a *nix host for Splunk App for Infrastructure and Manually configure metrics collection on a *nix host for Splunk App for Infrastructure.

When you set up the data collection agent on your host machine, and validate new hosts are connected, you can start monitoring your infrastructure. Hosts you are monitoring are called entities. Go to the Investigate page to monitor your entities in the Tile or List view. You can group your entities to monitor them more easily and further analyze your infrastructure by drilling down to the Overview Dashboard for entities only or the Analysis Workspace for entities and groups.

For information about stopping or removing the data collection agents, see Stop data collection on Splunk App for Infrastructure.

Prerequisites

Meet the following requirements to configure data collection:

Item	Requires
Linux/Unix machine	See Operating system support for data collection in the Install and Upgrade Splunk App for Infrastructure guide.
Data collection script	See *nix data collection requirements in the Install and Upgrade Splunk App for Infrastructure guide.
Administrator role	You must be an administrator to configure data collection.
HEC token	To create an HEC token, see Create an Event Collector token. You can also configure collectd to send data to the universal forwarder instead of using the HEC. For more information, see Send collectd data to a local universal forwarder.

Steps

Follow these steps to configure and use the data collection script so that the host sends metrics and log data to SAI.

1. Specify configuration options

Select data collection options for collecting metrics and logs from your host. If you're running SAI on Splunk Cloud, you must enter specific settings for the Monitoring machine, HEC port, and Receiver port, and set up the universal forwarder differently than the script deploys. For more information, see Install and configure the data collection agents on each applicable system in the Install and Upgrade Splunk App for Infrastructure guide.

1. In the SAI user interface, click the Add Data tab and select Linux/Unix.
2. Click Customize to select the metrics and log sources you want to collect data for. The cpu and uptime metrics are selected by default, and cannot be deselected.
• If you select cpu > Collect data for each CPU, metrics are stored for each CPU core, which enables you to split CPU usage by each core in the Analysis Workspace.
• If you select cpu > Collect sum over all CPUs, only aggregate metrics are stored for CPU usage.
3. When you are done selecting metrics and log sources, click Save.
4. Add Dimensions for easier troubleshooting, analysis, and filtering of entities. Dimensions are key/value pairs that provide metadata about the metric (describes the measurement) used for searching and filtering relevant datasets (distinct time series) during an investigation. Use the format of dimension:value, such as env:prod. The write_splunk collectd plug-in creates these five dimensions:
   • host
   • ip
   • os
   • os_version
   • kernel_version
   You cannot delete the dimensions that the plug-in creates.
5. Enter the Monitoring machine hostname or IP address of the machine that has SAI installed.
6. For HEC port, enter the port you use for the HTTP Event Collector (HEC) on the system you want to send metrics data to. Use port 8088 if it is available.
7. Enter the Receiver port of the machine you want to send log data to. Use port 9997 if it is available.
8. For Forwarder location, specify the directory where you want the script to install the universal forwarder.
9. Enter the HEC token of the machine you want to send data to.
10. Enable Authenticated Install to require the collectd repository signing key when the script installs collectd. This setting removes the --allow-unauthenticated flag and imports the repository's signing key, enabling you to verify the source location of the collectd package. This setting applies only when installing on the following operating systems:
    • Debian 7, 8
    • Ubuntu 14, 16
11. Enable Monitor Docker containers to collect metrics from Docker containers running on the host. Enable this option to track Docker containers you did not deploy with an orchestration tool such as Docker Swarm, Kubernetes, or OpenShift. Metrics for Docker containers are merged with the host system so that the host system and Docker containers data displays as one entity in SAI. For requirements about running the script with Docker monitoring enabled, see Docker (no orchestration) data collection requirements. If you want to monitor Docker containers you deployed with Kubernetes or OpenShift, see these topics:
    • Collect Kubernetes metrics and logs with Splunk App for Infrastructure
    • Collect OpenShift metrics and logs with Splunk App for Infrastructure
12. If you enabled Monitor Docker containers, enter the location of the Docker Socket. The default location of docker.sock is generally /var/run/. The Docker socket is the UNIX socket Docker listens to for Docker API calls.

2. Copy and paste the easy install script into the command line of your host

Deploy the script on your host to collect metrics and logs.

If you're running Ubuntu 18.04.1 LTS and haven't enabled the universe repository, the script may fail. Run these commands to enable the universe repository before running the script:

sudo apt-add-repository universe && sudo apt-get update

Follow these steps to deploy the script:

1. Open a terminal window on the monitoring machine.
2. Paste the script in the command line window.
3. Run the script. When you run the script for the first time, you may receive a message stating that the universal forwarder was installed without creating an admin user. If this occurs, you have to manually create admin credentials. For information about configuring user credentials, see user-seed.conf in the Splunk Enterprise Admin Manual.

3. Verify your data connection

Verify your data connection to start monitoring your infrastructure. When the script finishes running, the user interface indicates your host is connected and data is available to view.

It can take up to about five (5) minutes for your hosts to display in the user interface.

1. In the SAI user interface, click the Add Data tab.
2. If no new hosts are connected after a few minutes, click Refresh.
3. When new hosts are connected, click New host found to view your host.