Splunk® Enterprise

Metrics

Acrobat logo Download manual as PDF


Splunk Enterprise version 8.1 will no longer be supported as of April 19, 2023. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
Acrobat logo Download topic as PDF

Get metrics in from collectd

Collectd is an open source daemon that collects performance metrics from a variety of sources. Using the collectd write_http plugin, collectd sends metrics data to a data input in the Splunk platform using the HTTP Event Collector (HEC).

To send metrics using collectd, do the following:

  1. Configure the HTTP Event Collector (HEC) data input.
  2. Install collectd.
  3. Configure collectd.
  4. Start collectd.

Configure the HTTP Event Collector (HEC) data input

The HTTP Event Collector (HEC) is an endpoint that lets you send application events to your deployment of the Splunk platform using the HTTP or Secure HTTP (HTTPS) protocols. Configure this data input before setting up collectd because you'll need to use data input details for the collectd configuration.

  1. In Splunk Web, click Settings > Data Inputs.
  2. Under Local Inputs, click HTTP Event Collector.
  3. Verify that HEC is enabled.
    1. Click Global Settings.
    2. For All Tokens, click Enabled if this button is not already selected.
    3. Note the value for HTTP Port Number, which you'll need to configure collectd.
    4. Click Save.
  4. Configure an HEC token for sending data by clicking New Token.
  5. On the Select Source page, for Name, enter a token name, for example "collectd token".
  6. Leave the other options blank or unselected.
  7. Click Next.
  8. On the Input Settings page, for Source type, click Select.
  9. Click Select Source Type, then select Metrics > collectd_http.
  10. Next to Default Index, select your metrics index, or click Create a new index to create one.
    If you choose to create an index, in the New Index dialog box:
    1. Enter an Index Name. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. Index names cannot begin with an underscore or hyphen.
    2. For Index Data Type, click Metrics.
    3. Configure additional index properties as needed.
    4. Click Save.
  11. Click Review, and then click Submit.
  12. Copy the Token Value that is displayed, which you'll need to configure collectd.

Add collectd events directly to a metrics index

To test your data input, you can send collectd events directly to your metrics index using the /collector/raw REST API endpoint, which accepts data in the collectd JSON format. Your metrics index is assigned to an HEC data input that has its unique HEC token, and "collectd_http" as its source type.

The following example shows a curl command that sends a collectd event to the index associated with your HEC token:

curl https://localhost:8088/services/collector/raw?sourcetype=collectd_http   \
-H "Authorization: Splunk <HEC_token>"                                      \
-d '[{"values":[164.9196798931339196],"dstypes":["derive"],"dsnames":["value"],"time":1505356687.894,"interval":10.000,"host":"collectd","plugin":"protocols","plugin_instance":"IpExt","type":"protocol_counter","type_instance":"InOctets"}]'

You can verify the HEC data input is working by running a search using mcatalog to list all metric names, with the time range set to "All Time", for example:

| mcatalog values(metric_name) WHERE index=<your_metrics_index> AND metric_name=protocols.protocol_counter.InOctets.value

Or, use the Metrics Catalog REST endpoint to list metric names:

curl -u <admin:passwd> "https://localhost:8089/services/catalog/metricstore/metrics?earliest=0"

For more information about using HEC, see the following topics in Getting Data In:

See mstats and mcatalog in the Search Reference manual.

See the following topics in the REST API Reference Manual:

Install collectd

Install the collectd agent on the computers in your system from which you want to collect metrics.

  1. Go to the First steps page on the collectd website.
  2. Install collectd version 5.6 or higher, following the installation instructions for your operating system.

Configure collectd

The collectd server is an optional daemon that can be used to aggregate metrics from different inputs and one-to-many collectd clients.

Configure the collectd client to collect data by configuring plugins in the collectd.conf configuration file. The location of the collectd.conf file depends on your operating system. For details, see "Configuration" on the First steps page on the collectd website.

The write_http plugin

The write_http plugin requires the following fields from your HEC data input:

Field name Description Syntax Example
URL URL to which the values are submitted. This URL includes your Splunk host machine (IP address, host name, or load balancer name), and the HTTP port number. URL "https://<Splunk_host>:<HTTP_port>/services/collector/raw" URL "https://10.66.104.127:8088/services/collector/raw"
Header An HTTP header to add to the request. Header "Authorization: Splunk <HEC_token>" Header "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29"
Format The format of the data. Format "JSON" Format "JSON"

Enable and configure plugins

Enable each plugin below by uncommenting the plugin's LoadPlugin statement, then configure the plugin as described. Most of these plugins are for gathering basic OS-level metrics. The logfile plugin is needed for debugging purposes. You can configure additional plugins according to your requirements.

You might need to install some plugins separately, depending on your installation method and operating system. For details, see the collectd website.

Plugin Suggested configuration
cpu
LoadPlugin cpu
<Plugin cpu>
  ReportByCpu true
</Plugin>
interface
LoadPlugin interface

Use the default configuration.

load
LoadPlugin load
<Plugin load>
    ReportRelative true
</Plugin>
logfile
LoadPlugin logfile
<Plugin logfile>
    LogLevel info
    File STDOUT
    Timestamp true
    PrintSeverity false
</Plugin>
memory
LoadPlugin memory
<Plugin memory>
    ValuesAbsolute true
    ValuesPercentage true
</Plugin>
network
LoadPlugin network

Enable this plugin only if the collectd client is not on the same machine as the connectd server, then use the default configuration.

syslog
LoadPlugin syslog

Use the default configuration.

write_http You need the values from your HEC data input to configure this plugin.
LoadPlugin write_http
<Plugin write_http>
    <Node "node1">
        URL "https://<Splunk_host>:<HTTP_port>/services/collector/raw"
        Header "Authorization: Splunk <HEC_token>"
        Format "JSON"
        VerifyPeer false
        VerifyHost false
        Metrics true
        StoreRates true
    </Node>
</Plugin>

Start collectd

To start collectd, follow the instructions under "Starting the daemon" on the First steps page on the collectd website.

Modules for all of the enabled plugins in your collectd.conf file must be installed. Errors are displayed for any modules that are missing. For more about the available collectd plugins, see Table of Plugins on the collectd Wiki website.

Install modules according to your operating system. For example, on Linux you must install collectd-write_http.x86_64 to use the write_http plugin.

Tips:

  • For troubleshooting, refer to the collectd log file enabled by the logfile plugin for details.
  • Use the File setting in the logfile plugin to write to a specified file rather than to standard output. For example:
  • <Plugin logfile>
        LogLevel info
        File "/var/log/collectd.log"
        Timestamp true
        PrintSeverity false
    </Plugin>
  • If you are installing collectd on Linux, you can use yum to list available modules. For example, use this CLI command:
    yum list | grep collectd
  • In the collectd.conf file, set the FQDNLookup setting to false to render a friendly name for the domain name.
Last modified on 14 October, 2021
PREVIOUS
Configure StatsD dimension extraction through REST API endpoints
  NEXT
Get metrics in from other sources

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.6, 8.0.10, 7.2.10, 7.0.1, 8.0.5, 8.0.8, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.2.0, 9.2.1, 8.0.7, 8.0.9, 8.1.0


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters