Splunk® Enterprise

Getting Data In

Download manual as PDF

Download topic as PDF

Scale HTTP Event Collector with distributed deployments

Use the HTTP event collector (HEC) as part of a distributed Splunk platform deployment. The Splunk software processes HEC data in the same way as any other input.

To use HEC on a managed Splunk Cloud deployment, contact Splunk Support, and open a ticket to enable HEC. Cloudworks stacks have HEC enabled by default.

You should be familiar with distributed Splunk Enterprise deployment before proceeding. For more information about distributed deployments, see Distributed Splunk Enterprise overview in the Distributed Deployment Manual, and Components of a Splunk Enterprise deployment in the Capacity Planning Manual. For more information about deployment server, see About deployment server and forwarder management in the Updating Splunk Enterprise Instances manual.

Where to place the HEC

The HEC can be placed on heavy forwarders or indexers. The HEC, like any data input configuration, must reside on the component when the data enters the system. As a best practice, that is a heavy forwarder. HEC is not supported on universal forwarders. As with any data input configuration, you can also place the HEC directly on a clustered or non-clustered indexer, if necessary.

The deployment server will distribute any app that contains your HEC configurations. This configuration information includes:

  • HTTP Event Collector default values (port, SSL, source type, index)
  • SSL settings
  • HTTP Event Collector tokens

Each HEC input entry must contain a valid UUID for the token. The HEC stores its configurations in the $SPLUNK_HOME/etc/apps/splunk_httpinput/ directory (%SPLUNK_HOME%\etc\apps\splunk_httpinput\ on Windows).

Place and distribute the HEC on heavy forwarders

Use HEC in a distributed Splunk platform deployment that uses forwarders. Use a deployment server to distribute HEC configurations to the heavy forwarders in your Splunk platform deployment.

If you plan to distribute HEC configurations through the deployment server, set the useDeploymentServer option in the [http] stanza of inputs.conf on the deployment server to 1. When this option is set to 1 and you make UI-based HEC changes on the deployment server, those changes are placed directly in the $SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder, rather than in $SPLUNK_HOME/etc/apps/splunk_httpinput/. See the inputs.conf spec file for further information.

See Deploy a heavy forwarder in the Forwarding Data manual to learn more.

Place and distribute the HEC on non-clustered indexers

Use HEC with non-clustered indexers in one of two ways:

  • Place HEC on heavy forwarders that forward to the indexers, by using a deployment server to distribute configurations to the HEC.
  • Place HECs directly on the indexers, by using a deployment server to distribute configurations to the HEC.

See About deployment server and forwarder management in the Updating Splunk Enterprise Instances manual to learn more.

For more information about configuring deployment clients, see Configure deployment clients in the Splunk Enterprise manual.

Place and distribute the HEC on indexer cluster peer nodes

Use HEC in a distributed Splunk platform deployment that uses indexer clustering.

Use the configuration bundle method to distribute HEC configurations to the peer nodes. Using your HEC's port number, preferred protocol (HTTP or HTTPS), SSL settings, and HTTP Event Collector tokens, connect the HECs on your deployment's forwarders with the peer nodes of your deployment's indexer cluster. Tokens are managed centrally on the Splunk Enterprise instance running the cluster master.

See the Use forwarders to get data into the indexer cluster section in the Managing Indexers and Clusters of Indexers manual.

See the Update common peer configurations and apps topic in the Managing Indexers and Clustered Indexers manual.

Example serverclass.conf file

A server class is a group of deployment clients that you can manage as a single unit. You assign the deployment clients you want to use in your HTTP Event Collector deployment to one common server class. Later, when you distribute HTTP Event Collector settings to the deployment clients, only members of that server class will receive the configuration settings. The following example serverclass.conf file defines a server class "FWD2Local" for HTTP Event Collector.

[global]
whitelist.0=*
restartSplunkd=true
stateOnClient = enabled
 
[serverClass:FWD2Local]
whitelist.0=*
[serverClass:FWD2Local:app:splunk_httpinput]

For the purposes of deploying HTTP Event Collector settings, you can think of HEC as an app called "splunk_httpinput." Within the stanzas, you can set client filtering attributes and several non-filtering attributes.

For more information about available client filtering attributes, see the section Define filters through serverclass.conf in the topic Set up client filters. in the Updating Splunk Enterprise Instances Manual. To learn more about available non-filtering attributes, see the section what you can configure for a server class in the Use serverclass.conf to define server classes topic in the Updating Splunk Enterprise Instances Manual.

See also

For more information about distributed deployment, see Distributed Splunk Enterprise overview in the Distributed Deployment Manual, and Components of a Splunk Enterprise deployment in the Capacity Planning Manual.

For more information about distributed deployment, including advanced configuration options and general examples, see the Updating Splunk Instances Manual.

PREVIOUS
About HTTP Event Collector Indexer Acknowledgment
  NEXT
Format events for HTTP Event Collector

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 8.0.0


Comments

Step 4 in the distributed deployment steps mentions that when useDeploymentServer is set to true, it will use the $SPLUNK_HOME/etc/apps/splunk_httpinput/ folder for storing configuration. That seems incorrect. It will use the $SPLUNK_HOME/etc/deployment-apps/splunk_httpinput/ folder (which is exactly what you want, if you want to use the deployment server to manage HEC config on deployment clients).

FrankVl
June 7, 2018

Thank you for pointing that out, Strive. We have fixed the link.

Cgales splunk, Splunker
March 20, 2018

Hi,

Currently the link address of the hyperlink 'Configure an NGINX load balancer for HTTP Event Collector.' is http://docs.splunk.com/Documentation/Splunk/7.0.2/Data/dev.splunk.com/view/event-collector/SP-CAAAE9Q and it doesn't work. It should be http://dev.splunk.com/view/event-collector/SP-CAAAE9Q

Strive
March 20, 2018

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters