Get data into an Edge Processor using HTTP Event Collector

Edge Processors support the HTTP Event Collector (HEC). HEC is a mechanism that allows HTTP clients and logging agents to send data to the Splunk platform over HTTP or HTTPS. You can use HEC to send data programmatically and without requiring forwarders. If you already use HEC to ingest data in the Splunk platform, you can update your data ingestion workflow to use the Edge Processor solution. Send events from your HTTP clients and logging applications to an Edge Processor so that you can process your data before sending it to Splunk Enterprise, Splunk Cloud Platform, or other supported data destinations.

Edge Processors support the services/collector and services/collector/raw endpoints only. For more information about these HEC endpoints, see Usage information and examples for supported HEC endpoints in this topic and Input endpoint descriptions in the Splunk Enterprise REST API Reference Manual.

To get data from an HTTP client or logging agent into an Edge Processor using HEC, you must do the following:

1. Configure your Edge Processor to be able to receive data through HEC. See Activate the HEC receiver on your Edge Processor.
2. (Optional) Configure HEC token authentication in your Edge Processor to secure your HEC receiver. See Configure HEC token authentication in the Edge Processor service.
3. Configure your HTTP client or logging agent to send data to your Edge Processor by making HTTP POST requests to one of the supported HEC endpoints. See Send data to an Edge Processor using HEC.

Prerequisites

Before configuring your Edge Processor to receive data through HEC, complete the following tasks:

• In the Edge Processor service, make sure that the shared Edge Processor settings specify an appropriate port for receiving HEC data. By default, port 8088 is used. To verify or change the designated port, navigate to the Edge Processors page and then select Shared settings. The port number is specified in the HTTP Event Collector section of the Edge Processor shared settings page. See Configure shared Edge Processor settings for more information.
• On the host machine of your Edge Processor, make sure that the port for receiving HEC data is available and your network policy allows that port to be opened.
• If the HTTP requests from your data source include a HEC token, turn off the Enable indexer acknowledgement setting on that token.
• If the HTTP requests from your data source include a HEC token and you want to use it for authorization, make sure to turn on token authentication and add that token in the HTTP Event Collector section of the Edge Processor shared settings page. See Configure HEC token authentication in the Edge Processor service and Authorization header in this topic for more information.
• If you want to secure communications between your data source and your Edge Processor using mutually authenticated TLS (mTLS), which means that the data source and the Edge Processor must prove their identities by presenting valid TLS certificates before they can connect and communicate with each other, then obtain the following certificates in Privacy Enhanced Mail (PEM) format:
  • A client certificate, CA certificate, and private key that the data source can use to prove its identity.
  • A server certificate, CA certificate, and private key that the Edge Processor can use to prove its identity.

  These certificates can be self-signed or they can be signed by a third-party. See Obtain TLS certificates for data sources and Edge Processors for information on generating client and server certificates.

Activate the HEC receiver on your Edge Processor

Configure your Edge Processor to support incoming data that is transmitted through HEC.

The following instructions mention only the settings that are required for receiving HEC data. For information about other Edge Processor settings, see Add an Edge Processor.

1. In the Edge Processor service, navigate to the Edge Processors page.
2. To access the receiver settings, do one of the following:
   • If you want to configure an existing Edge Processor, find the row that lists that Edge Processor. Then, select the Actions icon () and select Edit Edge Processor.
   • If you want to configure a new Edge Processor, select New Edge Processor.
3. In the Receive data from these inputs section, select HTTP Event Collector.
4. (Optional) If you want to use mTLS to secure the communications between your HEC data sources and the Edge Processor, then do the following:
   1. In the Use mTLS with these inputs section, select HTTP Event Collector.
   2. If this is the first time that the mTLS settings have been turned on for this Edge Processor, then upload PEM files containing the certificates for proving the Edge Processor's identity in the Server private key, Server certificate, and CA certificates fields.

      The Edge Processor uses the same PEM files to prove its identity to all data sources where mTLS is used. For example, if you also select Splunk forwarders in the Use mTLS with these inputs section, then the Edge Processor uses the same server-side PEM files when receiving data from forwarders and HEC data sources.

5. Select Save.

The Edge Processor can now receive data from HTTP clients and logging agents using HEC. Next, you can configure HEC token authentication in your Edge Processor to secure your HEC receiver or move on to configure your HTTP clients and logging agents to send data to the Edge Processor.

Configure HEC token authentication in the Edge Processor service

If you want to secure your HEC receivers in Edge Processor by using HEC tokens, configure your shared Edge Processor settings to turn on token authentication.

The following instructions mention only the settings that are related to HEC token authentication. For information about other shared Edge Processor settings, see Configure shared Edge Processor settings.

1. To secure the HEC receiver in your Edge Processors by requiring incoming HTTP requests to be authenticated using a HEC token, do the following:
1. In the Token authentication section, select Edit.
2. Turn on Token authentication.
3. In the HEC tokens field, enter one or more tokens separated by commas, then select Add.
   
   Make sure to add the correct HEC token. If an invalid HEC token is used for token authentication, there could be possible data loss during the time of data ingestion. See Troubleshoot the Edge Processor solution for more information.
2. Select Save.

The HEC tokens in the shared Edge Processor settings are not synced to the HEC tokens in the Splunk platform. If a HEC token value is updated in the Splunk platform, then make sure to also update it the shared Edge Processor settings.

Send data to an Edge Processor using HEC

Configure your HTTP client or logging agent to make HTTP POST requests to a supported HEC endpoint on an Edge Processor instance. If your Edge Processor has multiple instances, then you must make the HTTP requests to either a specific instance or a load balancer that is configured to pass incoming requests to multiple instances.

The examples on this page use the curl command to generate the request, but you can use any tool or application that is compatible with the HTTP and REST specifications.

For information about the supported HEC endpoints, including the use case of each endpoint and examples that demonstrate how requests to different endpoints result in different data ingestion behavior by the Edge Processor, see Usage information and examples for supported HEC endpoints in this topic.

Depending on which HEC endpoint you're using, you need to format your HTTP requests differently. See Requirements and best practices for requests to HEC endpoints in Edge Processors in this topic.

Usage information and examples for supported HEC endpoints

You can direct the HTTP requests to the following supported HEC endpoints in Edge Processors:

• services/collector
• services/collector/raw

Using the services/collector endpoint in Edge Processors

Use the services/collector endpoint to send events in JSON format.

The Edge Processor treats each top-level JSON object as a distinct event, so you don't need to configure any additional event breaking behavior in the Edge Processor. Additionally, if the JSON object contains the following parameters, the Edge Processor extracts them into event fields: fields, host, index, source, sourcetype, and time. For more information about these parameters, see services/collector in the Splunk Enterprise REST API Reference Manual. If there are other data values that you want to extract into event fields, you'll need to configure field extractions in an Edge Processor pipeline or in the Splunk platform. For more information, refer to the resources listed in the See also section.

The following is an example of a request to the services/collector endpoint:

curl "http://buttercupgames.com:8088/services/collector" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}{"event": "something happened", "sourcetype": "manual", "host": "host_1.splunk.com", "fields":{"severity":"INFO", "category":["foo","bar"]}}'

This request contains 2 top-level JSON objects, so the Edge Processor ingests the request as 2 separate events. You do not need to configure any additional event breaking in the Edge Processor.

The first event contains these 3 fields:

_raw	sourcetype	host
Hello, world!	manual	host_1.splunk.com

The second event contains these 5 fields:

_raw	sourcetype	host	severity	category
something happened	manual	host_1.splunk.com	INFO	"foo", "bar"

Using the services/collector/raw endpoint in Edge Processors

Use the services/collector/raw endpoint to send raw data that requires event breaking.

The Edge Processor treats the raw text string as the body of a single event. If the string contains multiple events, you must configure an appropriate source type in the Edge Processor service so that the Edge Processor can break the raw data into distinct events.

If the request includes the following query string parameters, the Edge Processor extracts them into event fields: host, index, source, sourcetype, and time. For more information about these parameters, see services/collector/raw in the Splunk Enterprise REST API Reference Manual. If there are other data values that you want to extract into event fields, you'll need to configure field extractions in an Edge Processor pipeline or in the Splunk platform. For more information, refer to the resources listed in the See also section.

The following is an example of a request to the services/collector/raw endpoint:

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d 'Hello, world!
something happened'

Without event breaking configurations, the Edge Processor would ingest the request as a single event where the _raw field contains Hello, world! something happened.

To correctly break the raw data into 2 events, you must configure a source type in the Edge Processor service that treats newline characters as the delimiter between distinct events. Use these settings in the source type definition:

• Name: manual
• Line breaking: ([\r\n]+)

With this event breaking configuration, the Edge Processor ingests the request as 2 separate events.

The first event contains these 3 fields:

_raw	sourcetype	host
Hello, world!	manual	host_1.splunk.com

The second event contains these 3 fields:

_raw	sourcetype	host
something happened	manual	host_1.splunk.com

Requirements and best practices for requests to HEC endpoints in Edge Processors

Make sure that your HTTP requests follow the requirements and best practices described in the following sections:

• Event metadata
• URI format
• Authorization header
• Request body format for the services/collector endpoint
• Request body format for the services/collector/raw endpoint
• mTLS requirements

Event metadata

When sending data to an Edge Processor through a HEC endpoint, make sure to specify a sourcetype value. If an event is not associated with a source type, then Edge Processor pipelines cannot select that event for processing.

Additionally, as a best practice for reducing inaccuracies in the Edge Processor metrics, make sure that you specify a host value. The Inbound data sources metric displayed in the Edge Processor service is based on the number of distinct host values detected in the inbound events.

If you are sending data through the services/collector endpoint, you must specify the sourcetype and host values in the body of the HTTP request. However, if you are sending data through the services/collector/raw endpoint, then you must specify the sourcetype and host values in the Uniform Resource Indicator (URI) of the HTTP request.

URI format

The HTTP request must be directed to a URI written in this format:

<protocol>://<host>:<port>/<endpoint>?<parameter>

The variables are defined as follows:

• <protocol> is either http or https.
• <host> is the host name of a specific Edge Processor instance or a load balancer.
• <port> is the number of the port that the Edge Processor instance uses to receive HEC requests.

  This port number is a shared configuration setting that is specified in the Edge Processor service. See Configure shared Edge Processor settings for more information.

• <endpoint> is the HEC endpoint that you want to send the request to.
• <parameter> is one or more query string parameters that specify metadata for the event. For example, if you include the parameter sourcetype=manual, then the events that you send using this URI will have a sourcetype field containing the value manual.
  • The services/collector endpoint in Edge Processors does not support query string parameters.
  • The services/collector/raw endpoint in Edge Processors supports these parameters: host, index, source, sourcetype, and time. For more information about these parameters, see services/collector/raw in the Splunk Enterprise REST API Reference Manual.
  • To include multiple parameters, separate each parameter with an ampersand ( & ).
  • If you don't include any parameters, omit the question mark ( ? ) from the URI.

Query string parameters are typically optional, but they are mandatory when you send data to the services/collector/raw endpoint of an Edge Processor. You must use query string parameters to specify sourcetype and host values for your events.

Authorization header

When you send data to an Edge Processor using HEC, you can turn on token authentication to secure your HEC receivers in Edge Processor by using HEC tokens. This is optional. If token authentication is turned on and one or more tokens are added, then data sources can only send data to the Edge Processor if the HTTP request header includes a matching HEC token. To use token authentication, do the following:

• Configure your Edge Processor shared settings to turn on token authentication and add your token. See Configure HEC token authentication in the Edge Processor service for more information.
• Insert the token you added in the shared settings into the authorization header of your HTTP request. See Get HTTP request examples with your Edge Processor information for more information.

The authorization header appears after the URI in the HTTP request in this format:
-H "Authorization: Splunk <insert_auth-token>"

If your Edge Processor receives data that is not associated with a HEC token, and it then processes and sends that data to the Splunk platform using HEC, the Edge Processor uses the default HEC token specified in the pipeline destination. For more information about configuring a pipeline destination that uses HEC, see Send data from Edge Processors to non-connected Splunk platform deployments using HEC.

As a best practice for reducing the amount of configuration overrides in your HEC data ingestion setup, when token authentication is off in the Edge Processor settings, omit the Authorization header and HEC token from your inbound HTTP requests. Omitting the token during data ingestion allows you to rely on the pipeline destination configuration to specify the HEC token to be used when sending data out to a destination.

Request body format for the services/collector endpoint

When you send data to the services/collector endpoint, the body of the HTTP request must meet the following requirements:

• The request body is a JSON object.
• The JSON object must include the following:
  • An event parameter that contains the data being sent by your HTTP client or logging agent.
  • A sourcetype parameter that contains a value for the sourcetype metadata field in the event.
  • A host parameter that contains a value for the host metadata field in the event.

The following optional parameters are also supported: fields, index, and source. For more information, see services/collector in the Splunk Enterprise REST API Reference Manual.

For example, the following is a basic HTTP request that sends Hello, world! to an Edge Processor instance through the services/collector endpoint, sets the sourcetype parameter to manual, and sets the host parameter to host_1.splunk.com:

curl "http://buttercupgames.com:8088/services/collector" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

The following is the same example with an authorization header:

curl "http://buttercupgames.com:8088/services/collector" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

Request body format for the services/collector/raw endpoint

When you send data to the services/collector/raw endpoint, the body of the HTTP request must be a text string that contains the data being sent by your HTTP client or logging agent.

For example, the following is a basic HTTP request that sends Hello, world! to an Edge Processor instance through the services/collector/raw endpoint. This request uses query string parameters in the URI to specify required metadata such as sourcetype and host.

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" \
-d 'Hello, world!'

The following is the same example with an authorization header:

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB" \
-d 'Hello, world!'

The services/collector/raw endpoint interprets the request body as a raw text string even if the string contains formatting syntax. For example, the following request sends {"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"} instead of Hello, world! to the Edge Processor instance.

curl "http://buttercupgames.com:8088/services/collector/raw?sourcetype=manual&host=host_1.splunk.com" \
-d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

mTLS requirements

If your Edge Processor is configured to use mTLS to secure its communications with HEC data sources, then the HTTP requests must include PEM files containing the appropriate client certificate, CA certificate, and private key for proving the identity of the data source. Depending on how your HTTP client or logging agent makes requests, there can be a variety of different ways to include these PEM files.

For example, when using a curl command to make the HTTP request, you can use the --cert, --key, and --cacert flags to specify the PEM files:

curl --cert myClientCert.pem --key myPrivateKey.pem --cacert myCACert.pem --location "https://buttercupgames:8088/services/collector" -H "Authorization: Splunk 12345678-1234-1234-1234-1234567890AB"
\ -d '{"event": "Hello, world!", "sourcetype": "manual", "host": "host_1.splunk.com"}'

Get HTTP request examples with your Edge Processor information

In the Edge Processor service, you can find examples of basic HTTP requests that have the host name and port values relevant to your Edge Processors. You can use these examples as a starting point for your HTTP requests.

1. In the Edge Processor service, navigate to the Edge Processors page.
2. In the row that lists your Edge Processor, select the Actions icon () and then select Configure data sources.
3. In the Configure data sources side panel, use the drop-down list to select HTTP Event Collector.
4. If your Edge Processor has multiple instances, then do one of the following:
   • If you want to send data to multiple Edge Processor instances through a load balancer, select Load balancer.
   • If you want to send data to a specific Edge Processor instance, select Instance and then use the Select an instance drop-down list to select your desired instance.
5. To copy an example, select the Copy to clipboard icon () in the field that displays the example.

See also

You can configure an Edge Processor to send data to Splunk Enterprise or Splunk Cloud Platform using HEC. See Send data from Edge Processors to non-connected Splunk platform deployments using HEC.

If the Edge Processor does not extract all the event fields that you want when ingesting HEC data, you can configure pipelines to extract specific fields. Additionally, if you are sending data from an Edge Processor to Splunk Enterprise or Splunk Cloud Platform, you can rely on the automatic field extractions performed by the Splunk platform deployment or configure custom field extractions in the Splunk platform deployment.

• For information about using an Edge Processor to extract fields before sending your data to a destination, see Extract fields from event data using an Edge Processor and Extract timestamps from event data using an Edge Processor.
• For information about field extractions in the Splunk platform, see the following documentation:
  • When Splunk software extracts fields in the Splunk Cloud Platform Knowledge Manager Manual.
  • The Configure timestamps chapter and the Configure indexed field extraction chapter in the Splunk Cloud Platform Getting Data In manual.