Get data into an Edge Processor using HTTP Event Collector

Edge Processors support the HTTP Event Collector (HEC). HEC is a mechanism that allows HTTP clients and logging agents to send data to the Splunk platform over HTTP or HTTPS. You can use HEC to send data programmatically and without requiring forwarders. If you already use HEC to ingest data in the Splunk platform, you can update your data ingestion workflow to use the Edge Processor solution. Send events from your HTTP clients and logging applications to an Edge Processor so that you can process your data before sending it to Splunk Enterprise, Splunk Cloud Platform, or other supported data destinations.

Edge Processors support the services/collector and services/collector/raw endpoints only. For more information about these HEC endpoints, see Usage information and examples for supported HEC endpoints in this topic and Input endpoint descriptions in the Splunk Enterprise REST API Reference Manual.

To get data from an HTTP client or logging agent into an Edge Processor using HEC, you must do the following:

1. Configure your Edge Processor to be able to receive data through HEC. See Activate the HEC receiver on your Edge Processor.
2. (Optional) Configure HEC token authentication in your Edge Processor to secure your HEC receiver. See Configure HEC token authentication in the Edge Processor service.
3. Configure your HTTP client or logging agent to send data to your Edge Processor by making HTTP POST requests to one of the supported HEC endpoints. See Send data to an Edge Processor using HEC.

Prerequisites

Before configuring your Edge Processor to receive data through HEC, complete the following tasks:

• In the Edge Processor service, make sure that the shared Edge Processor settings specify an appropriate port for receiving HEC data. By default, port 8088 is used. To verify or change the designated port, navigate to the Edge Processors page and then select Shared settings. The port number is specified in the HTTP Event Collector section of the Edge Processor shared settings page. See Configure shared Edge Processor settings for more information.
• On the host machine of your Edge Processor, make sure that the port for receiving HEC data is available and your network policy allows that port to be opened.
• If the HTTP requests from your data source include a HEC token, turn off the Enable indexer acknowledgement setting on that token.
• If the HTTP requests from your data source include a HEC token and you want to use it for authorization, add that token in the HTTP Event Collector section of the Edge Processor shared settings page. See Configure HEC token authentication in the Edge Processor service and Authorization header in this topic for more information.
• To secure communications between your data source and your Edge Processor using TLS or mutually authenticated TLS (mTLS), which means that the data source and the Edge Processor must prove their identities by presenting valid TLS certificates before they can connect and communicate with each other, do the following:
  • If you want to use mTLS then obtain the following certificates in Privacy Enhanced Mail (PEM) format:
    • A client certificate, CA certificate, and private key that the data source can use to prove its identity.
    • A server certificate, CA certificate, and private key that the Edge Processor can use to prove its identity.
  • If you want to use TLS, then you must have the following certificates in PEM format:
    • A server certificate, CA certificate, and private key that the Edge Processor can use to prove its identity.

  These certificates can be self-signed or they can be signed by a third-party. See Obtain TLS certificates for data sources and Edge Processors for information on generating client and server certificates.

Activate the HEC receiver on your Edge Processor

Configure your Edge Processor to support incoming data that is transmitted through HEC.

The following instructions mention only the settings that are required for receiving HEC data. For information about other Edge Processor settings, see Add an Edge Processor.

1. In the Edge Processor service, navigate to the Edge Processors page.
2. To access the receiver settings, do one of the following:
   • If you want to configure an existing Edge Processor, find the row that lists that Edge Processor. Then, select the Actions icon () and select Edit Edge Processor.
   • If you want to configure a new Edge Processor, select New Edge Processor.
3. In the Receive data from these inputs section, select HTTP Event Collector.
4. If you want to use TLS or mTLS to secure communications between this Edge Processor and the data sources that are sending data to it, then do the following:
   1. In the HTTP Event Collector section, select your preferred type of connection protocol.
   2. If you choose to use mTLS for your input, upload PEM files containing the certificates for proving the Edge Processor's identity in the Server private key, Server certificate, and CA certificates fields.
   3. If you choose to use TLS for your data input, upload PEM files containing the certificates for proving the Edge Processor's identity in the Server private key, and Server certificate fields.

   The Edge Processor uses the same PEM files to prove its identity to all data sources where TLS or mTLS is used. For example, if you select both Splunk forwarders and HTTP Event Collector as data inputs and use TLS or mTLS with these inputs, then the Edge Processor uses the same server-side PEM files when receiving data from forwarders and HEC data sources.

5. Select Save.

The Edge Processor can now receive data from HTTP clients and logging agents using HEC. Next, you can configure HEC token authentication in your Edge Processor to secure your HEC receiver or move on to configure your HTTP clients and logging agents to send data to the Edge Processor.

Configure HEC token authentication in the Edge Processor service

If you want to secure your HEC receivers in Edge Processor by using HEC tokens, configure your shared Edge Processor settings to turn on token authentication.

The following instructions mention only the settings that are related to HEC token authentication. For information about other shared Edge Processor settings, see Configure shared Edge Processor settings.

1. To secure the HEC receiver in your Edge Processors by requiring incoming HTTP requests to be authenticated using a HEC token, do the following:
1. In the Token authentication section, select New token.
2. In the Add HEC token section, enter your token value in the Token value field.
   
   Make sure to add the correct HEC token. If an invalid HEC token is used for token authentication, there could be possible data loss during the time of data ingestion. See Troubleshoot the Edge Processor solution for more information.




3. (Optional) In the Source field enter a source value that you want to assign to the data that is received through this HEC token.
4. (Optional) In the Source type field enter a sourcetype value that you want to assign to the data that is received through this HEC token.
2. Select Add.

When token authentication is turned on, data sources can only send data to the Edge Processor through HEC if the HTTP request includes a matching HEC token. The token authentication feature is activated when at least one HEC token is added. If you want to deactivate the token authentication feature, you must delete all added tokens. See How the Splunk platform uses HTTP Event Collector tokens to get data in in the Splunk Enterprise Getting Data In manual for more information.

The HEC tokens in the shared Edge Processor settings are not synced to the HEC tokens in the Splunk platform. If a HEC token value is updated in the Splunk platform, then make sure to also update it in the shared Edge Processor settings.

Send data to an Edge Processor using HEC

Configure your HTTP client or logging agent to make HTTP POST requests to a supported HEC endpoint on an Edge Processor instance. If your Edge Processor has multiple instances, then you must make the HTTP requests to either a specific instance or a load balancer that is configured to pass incoming requests to multiple instances.

The examples on this page use the curl command to generate the request, but you can use any tool or application that is compatible with the HTTP and REST specifications.

For information about the supported HEC endpoints, including the use case of each endpoint and examples that demonstrate how requests to different endpoints result in different data ingestion behavior by the Edge Processor, see Usage information and examples for supported HEC endpoints in this topic.

Depending on which HEC endpoint you're using, you need to format your HTTP requests differently. See Requirements and best practices for requests to HEC endpoints in Edge Processors in this topic.

Usage information and examples for supported HEC endpoints

You can direct the HTTP requests to the following supported HEC endpoints in Edge Processors:

• services/collector
• services/collector/raw

Using the services/collector endpoint in Edge Processors

Use the services/collector endpoint to send events in JSON format.

The Edge Processor treats each top-level JSON object as a distinct event, so you don't need to configure any additional event breaking behavior in the Edge Processor. Additionally, if the JSON object contains the following keys, the Edge Processor extracts them into event fields: fields, host, index, source, sourcetype, and time. For more information about these keys, see Event metadata in the Splunk Cloud Platform Getting Data In manual. If there are other data values that you want to extract into event fields, you'll need to configure field extractions in an Edge Processor pipeline or in the Splunk platform. For more information, refer to the resources listed in the See also section.

The following is an example of a request to the services/collector endpoint:

curl "http://buttercupgames.com:8088/services/collector" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}{"event": "something happened", "sourcetype": "manual", "host": "host_1.splunk.com", "fields":{"severity":"INFO", "category":["foo","bar"]}}'

This request contains 2 top-level JSON objects, so the Edge Processor ingests the request as 2 separate events. You do not need to configure any additional event breaking in the Edge Processor.

The first event contains these 3 fields:

_raw	sourcetype	host
Hello, world!	manual	host_1.splunk.com

The second event contains these 5 fields:

_raw	sourcetype	host	severity	category
something happened	manual	host_1.splunk.com	INFO	"foo", "bar"

Using the services/collector/raw endpoint in Edge Processors

Use the services/collector/raw endpoint to send raw data that requires event breaking.

The Edge Processor treats the raw text string as the body of a single event. If the string contains multiple events, you must configure an appropriate source type in the Edge Processor service so that the Edge Processor can break the raw data into distinct events.

If the request includes the following query string parameters, the Edge Processor extracts them into event fields: host, index, source, sourcetype, and time. For more information about these parameters, see services/collector/raw in the Splunk Enterprise REST API Reference Manual. If there are other data values that you want to extract into event fields, you'll need to configure field extractions in an Edge Processor pipeline or in the Splunk platform. For more information, refer to the resources listed in the See also section.

The following is an example of a request to the services/collector/raw endpoint:

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d 'Hello, world!
something happened'

Without event breaking configurations, the Edge Processor would ingest the request as a single event where the _raw field contains Hello, world! something happened.

To correctly break the raw data into 2 events, you must configure a source type in the Edge Processor service that treats newline characters as the delimiter between distinct events. Use these settings in the source type definition:

• Name: manual
• Line breaking: ([\r\n]+)

With this event breaking configuration, the Edge Processor ingests the request as 2 separate events.

The first event contains these 3 fields:

_raw	sourcetype	host
Hello, world!	manual	host_1.splunk.com

The second event contains these 3 fields:

_raw	sourcetype	host
something happened	manual	host_1.splunk.com

Requirements and best practices for requests to HEC endpoints in Edge Processors

Make sure that your HTTP requests follow the requirements and best practices described in the following sections:

• Event metadata
• URI format
• Authorization header
• Edge Processor acknowledgement
• Request body format for the services/collector endpoint
• Request body format for the services/collector/raw endpoint
• TLS and mTLS requirements

Event metadata

When sending data to an Edge Processor through a HEC endpoint, make sure to specify a sourcetype value. If an event is not associated with a source type, then Edge Processor pipelines cannot select that event for processing.

Additionally, as a best practice for reducing inaccuracies in the Edge Processor metrics, make sure that you specify a host value. The Inbound data sources metric displayed in the Edge Processor service is based on the number of distinct host values detected in the inbound events.

If you are sending data through the services/collector endpoint, you must specify the sourcetype and host values in the body of the HTTP request. However, if you are sending data through the services/collector/raw endpoint, then you must specify the sourcetype and host values in the Uniform Resource Indicator (URI) of the HTTP request.

URI format

The HTTP request must be directed to a URI written in this format:

<protocol>://<host>:<port>/<endpoint>?<parameter>

The variables are defined as follows:

• <protocol> is either http or https.
• <host> is the host name of a specific Edge Processor instance or a load balancer.
• <port> is the number of the port that the Edge Processor instance uses to receive HEC requests.

  This port number is a shared configuration setting that is specified in the Edge Processor service. See Configure shared Edge Processor settings for more information.

• <endpoint> is the HEC endpoint that you want to send the request to.
• <parameter> is one or more query string parameters that specify metadata for the event. For example, if you include the parameter sourcetype=manual, then the events that you send using this URI will have a sourcetype field containing the value manual.
  • The services/collector endpoint in Edge Processors does not support query string parameters.
  • The services/collector/raw endpoint in Edge Processors supports these parameters: host, index, source, sourcetype, and time. For more information about these parameters, see services/collector/raw in the Splunk Enterprise REST API Reference Manual.
  • To include multiple parameters, separate each parameter with an ampersand ( & ).
  • If you don't include any parameters, omit the question mark ( ? ) from the URI.

Query string parameters are typically optional, but they are mandatory when you send data to the services/collector/raw endpoint of an Edge Processor. You must use query string parameters to specify sourcetype and host values for your events.

Authorization header

When you send data to an Edge Processor using HEC, you can turn on token authentication to secure your HEC receivers in Edge Processor by using HEC tokens. This is optional. If token authentication is activated, then data sources can only send data to the Edge Processor if the HTTP request header includes a matching HEC token. To use token authentication, do the following:

• In your Edge Processor shared settings, add your HEC token. See Configure HEC token authentication in the Edge Processor service for more information.
• Insert the token you added in the shared settings into the authorization header of your HTTP request. See Get HTTP request examples with your Edge Processor information for more information.

The authorization header appears after the URI in the HTTP request in this format:
-H "Authorization: Splunk <insert_auth-token>"

If your Edge Processor receives data that is not associated with a HEC token, and it then processes and sends that data to the Splunk platform using HEC, the Edge Processor uses the default HEC token specified in the pipeline destination. For more information about configuring a pipeline destination that uses HEC, see Send data from Edge Processors to non-connected Splunk platform deployments using HEC.

As a best practice for reducing the amount of configuration overrides in your HEC data ingestion setup, when token authentication is off in the Edge Processor settings, omit the Authorization header and HEC token from your inbound HTTP requests. Omitting the token during data ingestion allows you to rely on the pipeline destination configuration to specify the HEC token to be used when sending data out to a destination.

Edge Processor acknowledgement

When you send data to an Edge Processor using HEC, you can use the Edge Processor acknowledgement feature to verify that the Edge Processor received the data. This is optional. If you want to use Edge Processor acknowledgement, you need to add a channel ID header to your HTTP request. See About channels and sending data in the Splunk Enterprise Getting Data In manual for more information.

The Edge Processor acknowledgement is not the same as the Splunk platform indexer acknowledgement. The indexer acknowledgment feature verifies if the data was successfully indexed.

The channel ID header in your HTTP request needs to be in this format. Insert your channel ID in the <insert_channel_ID> placeholder:

-H "X-Splunk-Request-Channel: <insert_channel_ID>"

When you send the request, it returns an ackID value that you can use to confirm that the Edge Processor received your data. Each channel will have an ackID value unique to that specific channel only. The Edge Processor stores a maximum of 1,000,000 ackID values in memory that are not queried. Once the ackID value is queried, it will be removed. Once you exceed this limit, the Edge Processor removes the oldest ackID value. After receiving the ackID value, confirm that the Edge Processor received your data by sending a second HTTP request to the /services/collector/ack endpoint.

Send an HTTP request using the following format:

curl --location 'host_123ghnskfpi:10088/services/collector/ack?channel=<insert_channel_ID>'  -H 'Content-Type: application/json'  -d '{"acks":[<insert_ack_ID>]}'

Replace the following placeholders:

• <insert_channel_ID> with your channel ID.
• <insert_ack_ID> with the ackID value that you received from the first HTTP request.

This second request will return a value of either true or false depending on if the data was received by the Edge Processor. If the request returns a value of true, the Edge Processor has received your data and is written on the queue. If the request returns a value of false, the Edge Processor did not process and write your data on the queue or the ackID has already been removed due to reaching the maximum amount of ackID values in memory.

To see HTTP request examples with Edge Processor acknowledgment, see Get HTTP request examples with your Edge Processor information for more information.

Request body format for the services/collector endpoint

When you send data to the services/collector endpoint, the body of the HTTP request must meet the following requirements:

• The request body is a JSON object.
• The JSON object must include the following:
  • An event key that contains the data being sent by your HTTP client or logging agent.
  • A sourcetype key that contains a value for the sourcetype metadata field in the event.
  • A host key that contains a value for the host metadata field in the event.

The following optional keys are also supported: fields, index, source and time. For more information, see Event metadata in the Splunk Cloud Platform Getting Data In manual.

For example, the following is a basic HTTP request that sends Hello, world! to an Edge Processor instance through the services/collector endpoint, sets the sourcetype key to manual, and sets the host key to host_1.splunk.com:

curl "http://buttercupgames.com:8088/services/collector" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

The following is the same example with an authorization header:

curl "http://buttercupgames.com:8088/services/collector" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

Request body format for the services/collector/raw endpoint

When you send data to the services/collector/raw endpoint, the body of the HTTP request must be a text string that contains the data being sent by your HTTP client or logging agent.

For example, the following is a basic HTTP request that sends Hello, world! to an Edge Processor instance through the services/collector/raw endpoint. This request uses query string parameters in the URI to specify required metadata such as sourcetype and host.

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" \
-d 'Hello, world!'

The following is the same example with an authorization header:

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d 'Hello, world!'

The services/collector/raw endpoint interprets the request body as a raw text string even if the string contains formatting syntax. For example, the following request sends {"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"} instead of Hello, world! to the Edge Processor instance.

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

TLS and mTLS requirements

If your Edge Processor is configured to use TLS or mTLS to secure its communications with HEC data sources, then the HTTP requests must include PEM files containing the appropriate client certificate, CA certificate, and private key for proving the identity of the data source. Depending on how your HTTP client or logging agent makes requests, there can be a variety of different ways to include these PEM files.

For example, when using a curl command to make the HTTP request, you can use the --cert, --key, and --cacert flags to specify the PEM files:

curl --cert myClientCert.pem --key myPrivateKey.pem --cacert myCACert.pem --location "https://buttercupgames:8088/services/collector" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB"
\ -d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

Get HTTP request examples with your Edge Processor information

In the Edge Processor service, you can find examples of basic HTTP requests that have the host name and port values relevant to your Edge Processors. You can use these examples as a starting point for your HTTP requests.

1. In the Edge Processor service, navigate to the Edge Processors page.
2. In the row that lists your Edge Processor, select the Actions icon () and then select Configure data sources.
3. In the Configure data sources side panel, use the drop-down list to select HTTP Event Collector.
4. If you want to verify that the Edge Processor received your data, do the following:
   1. Select Edge Processor acknowledgement. A channel ID header is added to the example HTTP request.
   2. Enter your channel ID in the header.
      This request returns an ackID value that will be used to confirm that the Edge Processor received your data on step 7. See Edge Processor acknowledgement and About channels and sending data in the Splunk Enterprise Getting Data In manual for more information.
5. If your Edge Processor has multiple instances, then do one of the following:
   • If you want to send data to multiple Edge Processor instances through a load balancer, select Load balancer. If you are using a load balancer with the Edge Processor acknowledgement, ensure that you are using session "sticky sessions" when setting up your load balancer.
   • If you want to send data to a specific Edge Processor instance, select Instance and then use the Select an instance drop-down list to select your desired instance.
6. To copy an example, select the Copy to clipboard icon () in the field that displays the example.
7. To copy an example of the HTTP request for using Edge Processor acknowledgement and confirming that the data was received, do the following:
   1. Expand the Verify that the Edge Processor received the data section.
   2. Replace the <insert_channel_ID> placeholder with the ackID value that was returned in the initial response body.
   3. Select the Copy to clipboard icon () in the field that displays the example.

Precedence order of HEC tokens and metadata field values

You can secure your HEC receiver in your Edge Processors by requiring incoming HTTP requests to be authenticated using a HEC token.

Additionally, you can specify source and source type values for some of the metadata fields in the events.

Source field

When you send data out from your HEC data source to an Edge Processor, the value of the source field is determined based on the following precedence order:

1. The source value that is already specified in the event before the Edge Processor receives it.
2. The source value that is specified in the HEC token being used.

Source type field

When you send data out from your HEC data source to an Edge Processor, the value of the sourcetype field is determined based on the following precedence order:

1. The sourcetype value that is already specified in the event before the Edge Processor receives it.
2. The sourcetype value specified in the HEC token being used.

See also

You can configure an Edge Processor to send data to Splunk Enterprise or Splunk Cloud Platform using HEC. See Send data from Edge Processors to non-connected Splunk platform deployments using HEC.

If the Edge Processor does not extract the event fields or timestamps that you want when ingesting HEC data, you can configure pipelines to extract that information. Additionally, if you are sending data from an Edge Processor to Splunk Enterprise or Splunk Cloud Platform, you can rely on the automatic extractions performed by the Splunk platform deployment or configure custom extractions in the Splunk platform deployment.

• For information about using an Edge Processor to extract fields and timestamps before sending your data to a destination, see Extract fields from event data using an Edge Processor and Extract timestamps from event data using an Edge Processor.
• For information about field and timestamp extractions in the Splunk platform, see the following documentation:
  • When Splunk software extracts fields in the Splunk Cloud Platform Knowledge Manager Manual.
  • The Configure timestamps chapter and the Configure indexed field extraction chapter in the Splunk Cloud Platform Getting Data In manual.