Troubleshoot AWS Kinesis Firehose data ingestion

The AWS Kinesis Firehose delivery stream is responsible for sending the events to Splunk via the HTTP Event Collector(HEC) endpoint. Troubleshoot the AWS Kinesis Firehose data ingestion process.

AWS Kinesis Firehose data cannot be found

AWS Kinesis Firehose data cannot be found.

Cause

The AWS Kinesis Firehose is not configured correctly and data is not being ingested from the AWS Kinesis Firehose.

Solution

1. Log in to your AWS Account and navigate to the Amazon Kinesis Data Firehose console in the region.
2. Navigate to Delivery Streamsand locate the delivery stream for the data source with the name SplunkDM<data_source>DeliveryStream where data_source is one of CloudTrail, SecurityHub, GuardDuty, IAM Access Analyzer or CloudWatch logs.
3. For each AWS Firehose Delivery Stream that is created with Splunk templates, there is an S3 bucket configured to store the events that fail to be ingested to Splunk. You can check for failed events for this delivery stream.
   1. Click on the delivery stream and then click the Configuration tab. Navigate to the Backup settings section, and locate the S3 backup bucket link. The name of the bucket will start with splunkdmfailed-. Click on the S3 bucket link and check if there are any events in the S3 bucket. If there are events in the S3 bucket, it means the delivery stream attempted to send data to Splunk but failed. If there are no failed events in the backup S3 bucket and no logs in the log group for the delivery stream, it means the delivery stream did not send the data to Splunk.
4. Check the logs in the AWS CloudWatch LogGroup under the Destination error logs tab for that delivery stream. If you see any errors indicating problems related to HTTP Event Collector configuration, refer to Troubleshoot the HEC Configuration for more troubleshooting steps.
5. Check the Monitoring tab for the Delivery stream metrics. Make sure to adjust the time range and check for the Incoming records metrics section. If the Incoming records is 0 for the time range, then it means the delivery stream never got the events.
6. If the Incoming records is showing non-zero events, check the Bytes successfully processed by Lambda function metrics. If the values are 0, then check the logs in the CloudWatch Log group of the Lambda function under the Configuration > Transform records section. The logs will provide information about any exceptions that the Lambda function is encountering during runtime.
7. If the Delivery to Splunk success metrics is showing non-zero events but the events still don't show up Splunk indexer, Contact Splunk Support.