Configure the Splunk Add-on for Amazon Kinesis Firehose

Event formatting requirements

The Splunk Add-on for Amazon Kinesis Firehose supports data collection using raw and event HEC types. When you collect data using the raw endpoint, data is sent directly to the raw endpoint without any preprocessing.

When you collect data using an event endpoint, format your events into the JSON format expected by HEC before sending them from Amazon Kinesis Firehose to the Splunk platform. You can apply an AWS Lambda blueprint to preprocess your events into the JSON structure and set event-specific fields. This allows you greater control over how your events are handled by the Splunk platform. For example, you can create and apply a Lambda blueprint that sends data from the same Firehose stream to different indexes depending on event type. For information about using an AWS Lambda function to preprocess events into this format, see Use AWS Lambda with HTTP Event Collector on the Splunk Developer Portal.

Note: If you work with VPC Flow Log data, the aws:cloudwatchlogs:vpcflow contains a nested events JSON array that cannot be parsed by the HTTP Event Collector. In this case you can prepare this data for the Splunk platform using an AWS Lambda function that extracts the nested JSON events correctly into a newline-delimited set of events. For information about the required JSON structure, see Format events for HTTP Event Collector on the Splunk developer portal.

Indexers in AWS VPC

If you have placed your indexers on the AWS Virtual Private Cloud, use an elastic load balancer to send data to your indexers.

Indexers in Splunk Enterprise

If you have not placed your indexers in an AWS Virtual Private Cloud, but the indexers are accessible from AWS Firehose via public IPs, install a CA-signed SSL certificate on each indexer, then send data directly to your indexers.

Prepare your indexers before you proceed:

1. Install a CA-signed SSL certificate on each indexer.
2. Create a DNS name that resolves to the set of indexers that you plan to use to collect data from Amazon Kinesis Firehose. You will need this DNS name in a later step.

For more information about SSL, see the Additional resources topic at the end of this manual.

Configure Elastic Load Balancer settings

Configure an Elastic Load Balancer for the Splunk Add-on for Amazon Kinesis Firehose in an AWS Virtual Private Cloud.

ELBs are required. Application load balancers and network load balancers are not supported.

Create an elastic load balancer

Follow these steps to configure your ELB properly to receive data. For more detailed information about Elastic Load Balancers, see Elastic Load Balancing Documentation in the AWS documentation.

Prerequisites

• Amazon Kinesis Firehose requires the HEC endpoint to be terminated with a valid CA-signed SSL certificate. Import your valid CA-signed SSL certificates to AWS Certificate Manager or AWS IAM before creating or modifying your elastic load balancer. See Configure Security Settings in the AWS documentation.

Steps

1. Open the Amazon EC2 console.
2. On the navigation pane, under Load balancing, select Load Balancers.
3. Create a classic load balancer with the following parameters:

   Field in Amazon Web Services ELB UI	Value
   Select load balancer type	Classic load balancer
   Load balancer name	Name of your load balancer
   Load balancer protocol	HTTPS. Use the default or change the load balancer port.
   Assign or select a security group	The chosen security group needs to allow inbound traffic from load balancer to HTTP event collector port on indexers.
   Configure security settings	Select your CA-signed SSL certificate that you imported in the prerequisites step.
   Health Check settings	Ping protocol: HTTPS Ping port: 8088 Ping path: HTTPS:8088/services/collector/health/1.0 Timeout: 60 seconds Interval: 300 seconds Unhealthy threshold: 10 Healthy threshold: 2
   Add EC2 instances	Add all indexers that you are using to index data with this add-on.

4. Click Review and create, and verify in the following review page that your load balancer details are correct. After creating your elastic load balancer, modify the port configuration and the attributes as described below.

Modify an existing load balancer with the proper settings

Prerequisites

• An elastic load balancer that has been configured with the correct basic settings. This includes setting the load balancer protocol to HTTPS and uploading a valid CA-signed SSL certificate.
  

Steps

1. From the Load balancers page in the EC2 console, select your elastic load balancer with the basic settings already configured.
2. Modify your load balancer with the following parameters:

   Field in Amazon Web Services ELB UI	Value
   Health Check settings	Ping protocol: HTTPS Ping port: 8088 Ping path: HTTPS:8088/services/collector/health/1.0 Timeout: 60 seconds Interval: 300 seconds Unhealthy threshold: 10 Healthy threshold: 2
   Port configuration	Under Edit stickiness, select Enable load balancer generated cookie stickiness. Leave expiration period blank.
   Attributes	Under Edit idle timeout, enter 600 seconds.