About the Splunk HTTP Event Collector

The Splunk HTTP Event Collector (HEC) helps you send data to Splunk Enterprise and Splunk Cloud. HEC lets you send data, application logs, and metrics over HTTP (or HTTPS) directly to Splunk Enterprise or Splunk Cloud from your application. HEC operates with tokens, which means you do not need to embed Splunk Enterprise or Splunk Cloud credentials in your app or supporting files.

To get started with using HEC, see one of the following topics:

• Set up and use HTTP Event Collector in Splunk Web
• Set up and use HTTP Event Collector with configuration files
• Set up and use HTTP Event Collector from the CLI

How HTTP Event Collector works

The basics of using Splunk HTTP Event Collector follow:

1. Turn on HTTP Event Collector by enabling its endpoint. HEC is not enabled by default.
2. Generate an HEC token.
3. On the client that will log to HEC, create an HTTP POST request, and set its authentication header or key/value pair to include the HEC token.
4. POST data to the HEC token receiver.

For a walkthrough of HEC, see Use the HTTP Event Collector in Splunk Web.

You can send any kind of data to Splunk Enterprise and Splunk Cloud through HTTP Event Collector. Event data can be raw text or formatted within a JavaScript Object Notation (JSON) object. You can simplify the process by using one of the following logging libraries:

• Splunk logging for Java
• Splunk logging for JavaScript
• Splunk logging for .NET

These libraries automatically package and send data to HEC in the correct format. HEC supports assigning different sourcetypes, indexes, and groups of indexers ("output groups"), so you can fine-tune where and how your data gets consumed by Splunk Enterprise or Splunk Cloud. You can use a deployment server to deploy HTTP Event Collector configuration files.

Following is a sample event, formatted in JSON according to the HEC protocol. For more information about the contents of each event packet, see Format events for HTTP Event Collector.

{
    "time": 1426279439, 
    "host": "localhost",
    "source": "datasource",
    "sourcetype": "txt",
    "index": "main",
    "event": { "hello": "world" }
}

HTTP Event Collector data flow

HEC takes data in as follows:

1. On the data source, data is packaged using an agent such as:
   1. a Splunk logging library (Splunk logging for Java, Splunk logging for .NET, or Splunk logging for JavaScript)
   2. a JavaScript request library
   3. the Java Apache HTTP client
   4. some other client
2. Each HTTP request is assigned the same unique token in its authorization header (or auth key/value pair), which has been generated with the management endpoint on the Splunk Enterprise or Splunk Cloud instance, using any of the following:
   1. the HTTP Event Collector UI
   2. a cURL command
   3. the Splunk Enterprise command-line interface (CLI)
   4. (managed Splunk Cloud customers only) a Splunk Support ticket
3. The HTTP request, each of which includes the token, is sent to the appropriate Splunk Enterprise or Splunk Cloud endpoint.
4. The token is verified against the list of known good tokens. If it's valid, an affirmative (OK) response is returned to the sender and the data is accepted by Splunk Enterprise or Splunk Cloud.
5. Splunk Enterprise or Splunk Cloud sends the event data from the HTTP request to indexers to be indexed.

HTTP Event Collector workflow

There are three major workflows in HTTP Event Collector:

End user

An end user of HTTP Event Collector (most often a third-party app developer) simply needs to add a few lines of code to his or her app to enable it to log to HEC in Splunk Enterprise or Splunk Cloud. The easiest way to do this is to integrate the Splunk logging for Java, Splunk logging for JavaScript, or Splunk logging for .NET library into the app. If the user doesn't want to use one of the libraries, he or she must manually configure a mechanism to send event data over HTTP (or HTTPS) to the HTTP Event Collector REST API endpoint on the Splunk server.

Token admin

The token admin can be the Splunk Enterprise or e-commerce Splunk Cloud admin, or a different person who does not necessarily have experience with Splunk Enterprise or Splunk Cloud. Tokens are required for HTTP Event Collector to accept data that is sent to its port or endpoint. A token admin uses the Splunk Enterprise or Splunk Cloud management UI or Command Line Interface (CLI) to create, edit, disable, enable, and remove tokens. The token admin can also use the REST API token management endpoints to directly edit token configurations, and can enable or disable the HTTP Event Collector endpoints themselves.

Managed service Splunk Cloud customers must open a support ticket to administer HEC tokens.

Enabling HTTP Event Collector on Splunk Enterprise and e-commerce Splunk Cloud also adds a new capability: edit_token_http specifically enables roles to create and edit HTTP Event Collector tokens.

Splunk Enterprise or Splunk Cloud admin

On the Splunk Enterprise or e-commerce Splunk Cloud instance on which HTTP Event Collector is running, the admin can choose what do with and where to send the data that is sent from clients. For example, the admin can specify indexes, sourcetypes, and output groups. To do this, the admin edits the HTTP Event Collector endpoint.