Get metrics in from collectd
Collectd is an open source daemon that collects performance metrics from a variety of sources. Using the collectd write_http plugin, collectd sends metrics data to a data input in the Splunk platform using the HTTP Event Collector (HEC).
To send metrics using collectd, do the following:
- Configure the HTTP Event Collector (HEC) data input.
- Install collectd.
- Configure collectd.
- Start collectd.
Configure the HTTP Event Collector (HEC) data input
The HTTP Event Collector (HEC) is an endpoint that lets you send application events to your deployment of the Splunk platform using the HTTP or Secure HTTP (HTTPS) protocols. Configure this data input before setting up collectd because you'll need to use data input details for the collectd configuration.
- In Splunk Web, click Settings > Data Inputs.
- Under Local Inputs, click HTTP Event Collector.
- Verify that HEC is enabled.
- Click Global Settings.
- For All Tokens, click Enabled if this button is not already selected.
- Note the value for HTTP Port Number, which you'll need to configure collectd.
- Click Save.
- Configure an HEC token for sending data by clicking New Token.
- On the Select Source page, for Name, enter a token name, for example "collectd token".
- Leave the other options blank or unselected.
- Click Next.
- On the Input Settings page, for Source type, click Select.
- Click Select Source Type, then select Metrics > collectd_http.
- Next to Default Index, select your metrics index, or click Create a new index to create one.
If you choose to create an index, in the New Index dialog box:- Enter an Index Name. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. Index names cannot begin with an underscore or hyphen.
- For Index Data Type, click Metrics.
- Configure additional index properties as needed.
- Click Save.
- Click Review, and then click Submit.
- Copy the Token Value that is displayed, which you'll need to configure collectd.
Add collectd events directly to a metrics index
To test your data input, you can send collectd events directly to your metrics index using the /collector/raw
REST API endpoint, which accepts data in the collectd JSON format. Your metrics index is assigned to an HEC data input that has its unique HEC token, and "collectd_http" as its source type.
The following example shows a curl command that sends a collectd event to the index associated with your HEC token:
curl https://localhost:8088/services/collector/raw?sourcetype=collectd_http \ -H "Authorization: Splunk <HEC_token>" \ -d '[{"values":[164.9196798931339196],"dstypes":["derive"],"dsnames":["value"],"time":1505356687.894,"interval":10.000,"host":"collectd","plugin":"protocols","plugin_instance":"IpExt","type":"protocol_counter","type_instance":"InOctets"}]'
You can verify the HEC data input is working by running a search using mcatalog
to list all metric names, with the time range set to "All Time", for example:
| mcatalog values(metric_name) WHERE index=<your_metrics_index> AND metric_name=protocols.protocol_counter.InOctets.value
Or, use the Metrics Catalog REST endpoint to list metric names:
curl -u <admin:passwd> "https://localhost:8089/services/catalog/metricstore/metrics?earliest=0"
For more information about using HEC, see the following topics in Getting Data In:
- Set up and use HTTP Event Collector in Splunk Web
- Format events for the HTTP Event Collector
- Send metrics to a metrics index
See mstats and mcatalog in the Search Reference manual.
See the following topics in the REST API Reference Manual:
Install collectd
Install the collectd agent on the computers in your system from which you want to collect metrics.
- Go to the First steps page on the collectd website.
- Install collectd version 5.6 or higher, following the installation instructions for your operating system.
Configure collectd
The collectd server is an optional daemon that can be used to aggregate metrics from different inputs and one-to-many collectd clients.
Configure the collectd client to collect data by configuring plugins in the collectd.conf configuration file. The location of the collectd.conf file depends on your operating system. For details, see "Configuration" on the First steps page on the collectd website.
The write_http plugin
The write_http plugin requires the following fields from your HEC data input:
Field name | Description | Syntax | Example |
---|---|---|---|
URL | URL to which the values are submitted. This URL includes your Splunk host machine (IP address, host name, or load balancer name), and the HTTP port number. | URL "https://<Splunk_host>:<HTTP_port>/services/collector/raw" | URL "https://10.66.104.127:8088/services/collector/raw" |
Header | An HTTP header to add to the request. | Header "Authorization: Splunk <HEC_token>" | Header "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" |
Format | The format of the data. | Format "JSON" | Format "JSON" |
Enable and configure plugins
Enable each plugin below by uncommenting the plugin's LoadPlugin
statement, then configure the plugin as described. Most of these plugins are for gathering basic OS-level metrics. The logfile plugin is needed for debugging purposes. You can configure additional plugins according to your requirements.
You might need to install some plugins separately, depending on your installation method and operating system. For details, see the collectd website.
Plugin | Suggested configuration |
---|---|
cpu | LoadPlugin cpu <Plugin cpu> ReportByCpu true </Plugin> |
interface | LoadPlugin interface Use the default configuration. |
load | LoadPlugin load <Plugin load> ReportRelative true </Plugin> |
logfile | LoadPlugin logfile <Plugin logfile> LogLevel info File STDOUT Timestamp true PrintSeverity false </Plugin> |
memory | LoadPlugin memory <Plugin memory> ValuesAbsolute true ValuesPercentage true </Plugin> |
network | LoadPlugin network Enable this plugin only if the collectd client is not on the same machine as the connectd server, then use the default configuration. |
syslog | LoadPlugin syslog Use the default configuration. |
write_http | You need the values from your HEC data input to configure this plugin.
LoadPlugin write_http <Plugin write_http> <Node "node1"> URL "https://<Splunk_host>:<HTTP_port>/services/collector/raw" Header "Authorization: Splunk <HEC_token>" Format "JSON" VerifyPeer false VerifyHost false Metrics true StoreRates true </Node> </Plugin> |
Start collectd
To start collectd, follow the instructions under "Starting the daemon" on the First steps page on the collectd website.
Modules for all of the enabled plugins in your collectd.conf file must be installed. Errors are displayed for any modules that are missing. For more about the available collectd plugins, see Table of Plugins on the collectd Wiki website.
Install modules according to your operating system. For example, on Linux you must install collectd-write_http.x86_64 to use the write_http plugin.
Tips:
- For troubleshooting, refer to the collectd log file enabled by the logfile plugin for details.
- Use the
File
setting in the logfile plugin to write to a specified file rather than to standard output. For example:
<Plugin logfile> LogLevel info File "/var/log/collectd.log" Timestamp true PrintSeverity false </Plugin>
yum list | grep collectd
FQDNLookup
setting to false
to render a friendly name for the domain name.Configure StatsD dimension extraction through REST API endpoints | Get metrics in from other sources |
This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.0.9, 9.0.10, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.1.4, 9.1.5, 9.1.6, 9.2.0, 9.2.1, 9.2.2, 9.2.3, 9.3.0, 9.3.1, 8.1.0, 8.1.10, 8.1.11, 8.1.12
Feedback submitted, thanks!