Scale HTTP Event Collector with distributed deployments
Use the HTTP event collector (HEC) as part of a distributed Splunk platform deployment. The Splunk software processes HEC data in the same way as any other input.
To use HEC on a managed Splunk Cloud deployment, contact Splunk Support, and open a ticket to enable HEC. Cloudworks stacks have HEC enabled by default.
You should be familiar with distributed Splunk Enterprise deployment before proceeding. For more information about distributed deployments, see Distributed Splunk Enterprise overview in the Distributed Deployment Manual, and Components of a Splunk Enterprise deployment in the Capacity Planning Manual. For more information about deployment server, see About deployment server and forwarder management in the Updating Splunk Enterprise Instances manual.
Where to place the HEC
The HEC can be placed on heavy forwarders or indexers. The HEC, like any data input configuration, must reside on the component when the data enters the system. As a best practice, that is a heavy forwarder. HEC is not supported on universal forwarders. As with any data input configuration, you can also place the HEC directly on a clustered or non-clustered indexer, if necessary.
The deployment server will distribute any app that contains your HEC configurations. This configuration information includes:
- HTTP Event Collector default values (port, SSL, source type, index)
- SSL settings
- HTTP Event Collector tokens
Each HEC input entry must contain a valid UUID for the token. The HEC stores its configurations in the
$SPLUNK_HOME/etc/apps/splunk_httpinput/ directory (
%SPLUNK_HOME%\etc\apps\splunk_httpinput\ on Windows).
Place and distribute the HEC on heavy forwarders
Use HEC in a distributed Splunk platform deployment that uses forwarders. Use a deployment server to distribute HEC configurations to the heavy forwarders in your Splunk platform deployment.
If you plan to distribute HEC configurations through the deployment server, set the
useDeploymentServer option in the
[http] stanza of
inputs.conf on the deployment server to
1. When this option is set to
1 and you make UI-based HEC changes on the deployment server, those changes are placed directly in the
$SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder, rather than in
$SPLUNK_HOME/etc/apps/splunk_httpinput/. See the
inputs.conf spec file for further information.
See Deploy a heavy forwarder in the Forwarding Data manual to learn more.
Place and distribute the HEC on non-clustered indexers
Use HEC with non-clustered indexers in one of two ways:
- Place HEC on heavy forwarders that forward to the indexers, by using a deployment server to distribute configurations to the HEC.
- Place HECs directly on the indexers, by using a deployment server to distribute configurations to the HEC.
See About deployment server and forwarder management in the Updating Splunk Enterprise Instances manual to learn more.
For more information about configuring deployment clients, see Configure deployment clients in the Splunk Enterprise manual.
Place and distribute the HEC on indexer cluster peer nodes
Use HEC in a distributed Splunk platform deployment that uses indexer clustering.
Use the configuration bundle method to distribute HEC configurations to the peer nodes. Using your HEC's port number, preferred protocol (HTTP or HTTPS), SSL settings, and HTTP Event Collector tokens, connect the HECs on your deployment's forwarders with the peer nodes of your deployment's indexer cluster. Tokens are managed centrally on the Splunk Enterprise instance running the cluster master.
See the Use forwarders to get data into the indexer cluster section in the Managing Indexers and Clusters of Indexers manual.
See the Update common peer configurations and apps topic in the Managing Indexers and Clustered Indexers manual.
Example serverclass.conf file
A server class is a group of deployment clients that you can manage as a single unit. You assign the deployment clients you want to use in your HTTP Event Collector deployment to one common server class. Later, when you distribute HTTP Event Collector settings to the deployment clients, only members of that server class will receive the configuration settings. The following example
serverclass.conf file defines a server class "FWD2Local" for HTTP Event Collector.
[global] whitelist.0=* restartSplunkd=true stateOnClient = enabled [serverClass:FWD2Local] whitelist.0=* [serverClass:FWD2Local:app:splunk_httpinput]
For the purposes of deploying HTTP Event Collector settings, you can think of HEC as an app called "splunk_httpinput." Within the stanzas, you can set client filtering attributes and several non-filtering attributes.
For more information about available client filtering attributes, see the section Define filters through serverclass.conf in the topic Set up client filters. in the Updating Splunk Enterprise Instances Manual. To learn more about available non-filtering attributes, see the section what you can configure for a server class in the Use serverclass.conf to define server classes topic in the Updating Splunk Enterprise Instances Manual.
For more information about distributed deployment, see Distributed Splunk Enterprise overview in the Distributed Deployment Manual, and Components of a Splunk Enterprise deployment in the Capacity Planning Manual.
For more information about distributed deployment, including advanced configuration options and general examples, see the Updating Splunk Instances Manual.
About HTTP Event Collector Indexer Acknowledgment
Format events for HTTP Event Collector
This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 8.0.0