Configure the Splunk Add-on for Amazon Kinesis Firehose
Event formatting requirements
The Splunk Add-on for Amazon Kinesis Firehose supports data collection using raw and event HEC types. When you collect data using the raw endpoint, data is sent directly to the raw endpoint without any preprocessing.
When you collect data using an event endpoint, format your events into the JSON format expected by HEC before sending them from Amazon Kinesis Firehose to the Splunk platform. You can apply an AWS Lambda blueprint to preprocess your events into the JSON structure and set event-specific fields. This allows you greater control over how your events are handled by the Splunk platform. For example, you can create and apply a Lambda blueprint that sends data from the same Firehose stream to different indexes depending on event type. For information about using an AWS Lambda function to preprocess events into this format, see Use AWS Lambda with HTTP Event Collector on the Splunk Developer Portal.
Note: If you work with VPC Flow Log data, the aws:cloudwatchlogs:vpcflow
contains a nested events JSON array that cannot be parsed by the HTTP Event Collector. In this case you can prepare this data for the Splunk platform using an AWS Lambda function that extracts the nested JSON events correctly into a newline-delimited set of events. For information about the required JSON structure, see Format events for HTTP Event Collector on the Splunk developer portal.
Indexers in AWS VPC
If you have placed your indexers on the AWS Virtual Private Cloud, use an elastic load balancer to send data to your indexers.
Indexers in Splunk Enterprise
If you have not placed your indexers in an AWS Virtual Private Cloud, but the indexers are accessible from AWS Firehose via public IPs, install a CA-signed SSL certificate on each indexer, then send data directly to your indexers.
Prepare your indexers before you proceed:
- Install a CA-signed SSL certificate on each indexer.
- Create a DNS name that resolves to the set of indexers that you plan to use to collect data from Amazon Kinesis Firehose. You will need this DNS name in a later step.
For more information about SSL, see the Additional resources topic at the end of this manual.
Configure Elastic Load Balancer settings
Configure an Elastic Load Balancer for the Splunk Add-on for Amazon Kinesis Firehose in an AWS Virtual Private Cloud.
ELBs are required. Application load balancers and network load balancers are not supported.
Create an elastic load balancer
Follow these steps to configure your ELB properly to receive data. For more detailed information about Elastic Load Balancers, see Elastic Load Balancing Documentation in the AWS documentation.
Prerequisites
- Amazon Kinesis Firehose requires the HEC endpoint to be terminated with a valid CA-signed SSL certificate. Import your valid CA-signed SSL certificates to AWS Certificate Manager or AWS IAM before creating or modifying your elastic load balancer. See Configure Security Settings in the AWS documentation.
Steps
- Open the Amazon EC2 console.
- On the navigation pane, under Load balancing, select Load Balancers.
- Create a classic load balancer with the following parameters:
Field in Amazon Web Services ELB UI Value Select load balancer type Classic load balancer Load balancer name Name of your load balancer Load balancer protocol HTTPS. Use the default or change the load balancer port. Assign or select a security group The chosen security group needs to allow inbound traffic from load balancer to HTTP event collector port on indexers. Configure security settings Select your CA-signed SSL certificate that you imported in the prerequisites step. Health Check settings Ping protocol: HTTPS Ping port: 8088
Ping path: HTTPS:8088/services/collector/health/1.0
Timeout: 60 seconds
Interval: 300 seconds
Unhealthy threshold: 10
Healthy threshold: 2Add EC2 instances Add all indexers that you are using to index data with this add-on. - Click Review and create, and verify in the following review page that your load balancer details are correct. After creating your elastic load balancer, modify the port configuration and the attributes as described below.
Modify an existing load balancer with the proper settings
Prerequisites
- An elastic load balancer that has been configured with the correct basic settings. This includes setting the load balancer protocol to HTTPS and uploading a valid CA-signed SSL certificate.
Steps
- From the Load balancers page in the EC2 console, select your elastic load balancer with the basic settings already configured.
- Modify your load balancer with the following parameters:
Field in Amazon Web Services ELB UI Value Health Check settings Ping protocol: HTTPS Ping port: 8088
Ping path: HTTPS:8088/services/collector/health/1.0
Timeout: 60 seconds
Interval: 300 seconds
Unhealthy threshold: 10
Healthy threshold: 2Port configuration Under Edit stickiness, select Enable load balancer generated cookie stickiness. Leave expiration period blank.
Attributes Under Edit idle timeout, enter 600 seconds.
This documentation applies to the following versions of Splunk® Enterprise: 7.3.0, 7.3.1
Feedback submitted, thanks!