Collect *nix metrics and logs with the data collection script in ITSI

Use the data collection script to configure data collection agents on a *nix host you want to collect metrics and log data from. If you're running Docker containers without an orchestration tool like Kubernetes or OpenShift, you can also use the script to monitor Docker containers on Linux hosts as well.

The data collection script requires internet access. If you don't have internet access, configure data collection manually. For more information, see these topics:

• Manually collect metrics from a *nix host in ITSI
• Manually collect logs from a *nix host in ITSI

To collect data from a system running SELinux, see Collect data in ITSI with SELinux.

If you haven't seen the requirements yet, see *nix integration requirements for ITSI.

If you're using Splunk Cloud Platform, you need to enter specific information according to your cloud stack when you configure an integration. For more information, see Send data to Splunk Cloud Platform with ITSI data collection agents.

To see which data the *nix integration sends to ITSI, see *nix data you can collect with ITSI.

Prerequisites

Requirement	Description
*nix host	See *nix integration operating system support.
Dependencies	See Required *nix dependencies.
Administrator role	In Splunk Enterprise, you have to be a user with the admin role. In Splunk Cloud Platform, you have to be a user with the sc_admin role.
HEC token	See HTTP Event Collector to collect entity integration data in ITSI. Alternatively, you can configure collectd to send data to the local universal forwarder instead of using the HEC. For more information, see Send collectd data to a local universal forwarder.
Internet access	The data collection script downloads a universal forwarder package from https://www.splunk.com/en_us/download/universal-forwarder.html to collect logs, and downloads collectd to collect metrics. Where the data collection script downloads collectd from depends on your operating system. For more information about collectd install locations, see collectd package sources, install commands, and locations for ITSI.

Steps to configure the data collection script for *nix hosts

Follow these steps to configure and use the data collection script to collect *nix metrics and logs in ITSI.

1. Specify configuration options

Configure data collection options for collecting metrics and logs from your host.

1. From the ITSI main menu, go to Configuration > Data Integrations.
2. Select the Unix and Linux chicklet.
3. Select Collectd.
4. Click Customize to select the metrics and log sources you want to collect data for. The cpu and uptime metrics are selected by default, and cannot be deselected.
• If you select cpu > Collect data for each CPU, metrics are stored for each CPU core, which enables you to split CPU usage by each core in the Analysis Workspace.
• If you select cpu > Collect sum over all CPUs, only aggregate metrics are stored for CPU usage.
5. When you're done selecting metrics and log sources, click Save.
6. Add Dimensions for easier troubleshooting, analysis, and filtering of entities. Dimensions are key/value pairs associated with an entity that you can use for searching and filtering during an investigation. Use the format dimension:value, such as env:prod or region:uswest. You can't delete dimensions the plug-in creates.
7. Enter the Monitoring machine hostname or IP address of the Splunk Enterprise instance you want to send metrics and log data to.
8. For HEC port, enter the port you use for the HTTP Event Collector (HEC) on the Splunk Enterprise instance you want to send metrics data to.
9. Enter the Receiver port of the machine you want to send log data to.
10. For Forwarder location, specify the directory where you want the script to install the universal forwarder on the host.
11. For HEC token enter the value of the HEC token you configured to receive metrics data for ITSI entities.
12. Enable Authenticated Install to require the collectd repository signing key when the script installs collectd. This setting removes the --allow-unauthenticated flag and imports the repository's signing key, enabling you to verify the source location of the collectd package. This setting applies only when installing on the following operating systems:
    • Debian 7, 8
    • Ubuntu 14, 16
13. Enable Monitor Docker containers to collect metrics from Docker containers running on the host. This option is available for only Linux and hosts. Enable this option to monitor Docker containers on the host you didn't deploy with an orchestration tool such as Kubernetes or OpenShift. Metrics for Docker containers are merged with the host system so that the host system and Docker containers data displays as one entity.
14. If you enabled Monitor Docker containers for the Linux host, enter the location of the Docker Socket. The default location of docker.sock is generally /var/run/. The Docker socket is the UNIX socket Docker listens to for Docker API calls.

2. Copy and paste the data collection script in a command line on the host

Deploy the script on your host to collect metrics and logs.

If you're running Ubuntu 18.04.1 LTS and haven't enabled the universe repository, the script may fail. Run these commands to enable the universe repository before running the script:

sudo apt-add-repository universe && sudo apt-get update

Follow these steps to deploy the script:

1. Open a terminal window on the monitoring machine.
2. Paste the script in the command line window.
3. Run the script. When you run the script for the first time, you may receive a message stating that the universal forwarder was installed without creating an admin user. If this occurs, you have to manually create admin credentials. For information about configuring user credentials, see user-seed.conf in the Splunk Enterprise Admin Manual.

3. Verify your data connection

Verify your data connection to start monitoring your infrastructure. It can take up to about five minutes for your host to display in the user interface.

In the ITSI user interface, go to Configuration > Entity Management and wait for new hosts to start appearing. Each host has the entity type *nix.