Data Manager

Troubleshooting Manual

This documentation does not apply to the most recent version of Data Manager. For documentation on the most recent version, go to the latest release.

Troubleshoot AWS Kinesis Firehose data ingestion

The AWS Kinesis Firehose delivery stream is responsible for sending the events to Splunk via the HTTP Event Collector(HEC) endpoint. Troubleshoot the AWS Kinesis Firehose data ingestion process.

AWS Kinesis Firehose data cannot be found

AWS Kinesis Firehose data cannot be found.

Cause

The AWS Kinesis Firehose is not configured correctly and data is not being ingested from the AWS Kinesis Firehose.

Solution

  1. Log in to your AWS Account and navigate to the Amazon Kinesis Data Firehose console in the region.
  2. Navigate to Delivery Streamsand locate the delivery stream for the data source with the name SplunkDM<data_source>DeliveryStream where data_source is one of CloudTrail, SecurityHub, GuardDuty, IAM Access Analyzer or CloudWatch logs.
  3. For each AWS Firehose Delivery Stream that is created with Splunk templates, there is an S3 bucket configured to store the events that fail to be ingested to Splunk. You can check for failed events for this delivery stream.
    1. Click on the delivery stream and then click the Configuration tab. Navigate to the Backup settings section, and locate the S3 backup bucket link. The name of the bucket will start with splunkdmfailed-. Click on the S3 bucket link and check if there are any events in the S3 bucket. If there are events in the S3 bucket, it means the delivery stream attempted to send data to Splunk but failed. If there are no failed events in the backup S3 bucket and no logs in the log group for the delivery stream, it means the delivery stream did not send the data to Splunk.
  4. Check the logs in the AWS CloudWatch LogGroup under the Destination error logs tab for that delivery stream. If you see any errors indicating problems related to HTTP Event Collector configuration, refer to Troubleshoot the HEC Configuration for more troubleshooting steps.
  5. Check the Monitoring tab for the Delivery stream metrics. Make sure to adjust the time range and check for the Incoming records metrics section. If the Incoming records is 0 for the time range, then it means the delivery stream never got the events.
  6. If the Incoming records is showing non-zero events, check the Bytes successfully processed by Lambda function metrics. If the values are 0, then check the logs in the CloudWatch Log group of the Lambda function under the Configuration > Transform records section. The logs will provide information about any exceptions that the Lambda function is encountering during runtime.
  7. If the Delivery to Splunk success metrics is showing non-zero events but the events still don't show up Splunk indexer, Contact Splunk Support.
Last modified on 07 February, 2022
Troubleshoot AWS CloudWatch Log data ingestion   Troubleshoot AWS Lambda Functions data ingestion

This documentation applies to the following versions of Data Manager: 1.3.1


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters