Get metrics in from collectd

Collectd is an open source daemon that collects performance metrics from a variety of sources. Using the collectd write_http plugin, collectd sends metrics data to a data input in the Splunk platform using the HTTP Event Collector (HEC).

To send metrics using collectd, do the following:

1. Configure the HTTP Event Collector (HEC) data input.
2. Install collectd.
3. Configure collectd.
4. Start collectd.

Configure the HTTP Event Collector (HEC) data input

The HTTP Event Collector (HEC) is an endpoint that lets you send application events to your deployment of the Splunk platform using the HTTP or Secure HTTP (HTTPS) protocols. Configure this data input before setting up collectd because you'll need to use data input details for the collectd configuration.

1. In Splunk Web, click Settings > Data Inputs.
2. Under Local Inputs, click HTTP Event Collector.
3. Verify that HEC is enabled.
   1. Click Global Settings.
   2. For All Tokens, click Enabled if this button is not already selected.
   3. Note the value for HTTP Port Number, which you'll need to configure collectd.
   4. Click Save.
4. Configure an HEC token for sending data by clicking New Token.
5. On the Select Source page, for Name, enter a token name, for example "collectd token".
6. Leave the other options blank or unselected.
7. Click Next.
8. On the Input Settings page, for Source type, click Select.
9. Click Select Source Type, then select Metrics > collectd_http.
10. Next to Default Index, select your metrics index, or click Create a new index to create one.
    If you choose to create an index, in the New Index dialog box:
    1. Enter an Index Name. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. Index names cannot begin with an underscore or hyphen.
    2. For Index Data Type, click Metrics.
    3. Configure additional index properties as needed.
    4. Click Save.
11. Click Review, and then click Submit.
12. Copy the Token Value that is displayed, which you'll need to configure collectd.

Add collectd events directly to a metrics index

To test your data input, you can send collectd events directly to your metrics index using the /collector/raw REST API endpoint, which accepts data in the collectd JSON format. Your metrics index is assigned to an HEC data input that has its unique HEC token, and "collectd_http" as its source type.

The following example shows a curl command that sends a collectd event to the index associated with your HEC token:

curl https://localhost:8088/services/collector/raw?sourcetype=collectd_http   \
-H "Authorization: Splunk <HEC_token>"                                      \
-d '[{"values":[164.9196798931339196],"dstypes":["derive"],"dsnames":["value"],"time":1505356687.894,"interval":10.000,"host":"collectd","plugin":"protocols","plugin_instance":"IpExt","type":"protocol_counter","type_instance":"InOctets"}]'

You can verify the HEC data input is working by running a search using mcatalog to list all metric names, with the time range set to "All Time", for example:

| mcatalog values(metric_name) WHERE index=<your_metrics_index> AND metric_name=protocols.protocol_counter.InOctets.value

Or, use the Metrics Catalog REST endpoint to list metric names:

curl -u <admin:passwd> "https://localhost:8089/services/catalog/metricstore/metrics?earliest=0"

For more information about using HEC, see the following topics in Getting Data In:

• Set up and use HTTP Event Collector in Splunk Web
• Format events for the HTTP Event Collector
• Send metrics to a metrics index

See mstats and mcatalog in the Search Reference manual.

See the following topics in the REST API Reference Manual:

• Metrics Catalog endpoint descriptions
• /collector
• /collector/raw

Install collectd

Install the collectd agent on the computers in your system from which you want to collect metrics.

1. Go to the First steps page on the collectd website.
2. Install collectd version 5.6 or higher, following the installation instructions for your operating system.

Configure collectd

The collectd server is an optional daemon that can be used to aggregate metrics from different inputs and one-to-many collectd clients.

Configure the collectd client to collect data by configuring plugins in the collectd.conf configuration file. The location of the collectd.conf file depends on your operating system. For details, see "Configuration" on the First steps page on the collectd website.

The write_http plugin

The write_http plugin requires the following fields from your HEC data input:

Field name	Description	Syntax	Example
URL	URL to which the values are submitted. This URL includes your Splunk host machine (IP address, host name, or load balancer name), and the HTTP port number.	URL "https://<Splunk_host>:<HTTP_port>/services/collector/raw"	URL "https://10.66.104.127:8088/services/collector/raw"
Header	An HTTP header to add to the request.	Header "Authorization: Splunk <HEC_token>"	Header "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29"
Format	The format of the data.	Format "JSON"	Format "JSON"

Enable and configure plugins

Enable each plugin below by uncommenting the plugin's LoadPlugin statement, then configure the plugin as described. Most of these plugins are for gathering basic OS-level metrics. The logfile plugin is needed for debugging purposes. You can configure additional plugins according to your requirements.

You might need to install some plugins separately, depending on your installation method and operating system. For details, see the collectd website.

Plugin	Suggested configuration
cpu	LoadPlugin cpu <Plugin cpu> ReportByCpu true </Plugin>
interface	LoadPlugin interface Use the default configuration.
load	LoadPlugin load <Plugin load> ReportRelative true </Plugin>
logfile	LoadPlugin logfile <Plugin logfile> LogLevel info File STDOUT Timestamp true PrintSeverity false </Plugin>
memory	LoadPlugin memory <Plugin memory> ValuesAbsolute true ValuesPercentage true </Plugin>
network	LoadPlugin network Enable this plugin only if the collectd client is not on the same machine as the connectd server, then use the default configuration.
syslog	LoadPlugin syslog Use the default configuration.
write_http	You need the values from your HEC data input to configure this plugin. LoadPlugin write_http <Plugin write_http> <Node "node1"> URL "https://<Splunk_host>:<HTTP_port>/services/collector/raw" Header "Authorization: Splunk <HEC_token>" Format "JSON" VerifyPeer false VerifyHost false Metrics true StoreRates true </Node> </Plugin>

Start collectd

To start collectd, follow the instructions under "Starting the daemon" on the First steps page on the collectd website.

Modules for all of the enabled plugins in your collectd.conf file must be installed. Errors are displayed for any modules that are missing. For more about the available collectd plugins, see Table of Plugins on the collectd Wiki website.

Install modules according to your operating system. For example, on Linux you must install collectd-write_http.x86_64 to use the write_http plugin.

Tips:

• For troubleshooting, refer to the collectd log file enabled by the logfile plugin for details.
• Use the File setting in the logfile plugin to write to a specified file rather than to standard output. For example:

<Plugin logfile>
    LogLevel info
    File "/var/log/collectd.log"
    Timestamp true
    PrintSeverity false
</Plugin>

• If you are installing collectd on Linux, you can use yum to list available modules. For example, use this CLI command:
  

  yum list | grep collectd

• In the collectd.conf file, set the FQDNLookup setting to false to render a friendly name for the domain name.