Troubleshoot the Splunk Add-on for Amazon Kinesis Firehose

General troubleshooting

For troubleshooting tips that you can apply to all add-ons, see Troubleshoot add-ons in Splunk Add-ons. For additional resources, see Support and resource links for add-ons in Splunk Add-ons.

Cannot launch add-on

This add-on does not have views and is not intended to be visible in Splunk Web.

For more details about add-on visibility and instructions for turning visibility off, see Troubleshoot add-ons in Splunk Add-ons.

Firehose error exceptions

See Data Not Delivered to Splunk in the AWS documentation.

Data delivery errors

You can view the error logs related to Kinesis Firehose data delivery failure using the Kinesis Firehose console or CloudWatch console. See the Accessing CloudWatch Logs for Kinesis Firehose section in the Monitoring with Amazon CloudWatch Logs topic from the AWS documentation.

SSL-related data delivery errors

Amazon Kinesis Firehose requires HTTP Event Collector (HEC) endpoint to be terminated with a valid CA-signed certificate matching the DNS hostname used to connect to your HEC endpoint. If you are seeing an error message "Could not connect to the HEC endpoint. Make sure that the HEC endpoint URL is valid and reachable from Kinesis Firehose," then your SSL certificate might not be valid.

Test if your SSL certificate is valid by opening your HEC endpoint in a web browser. If you are using a self-signed certificate, you will receive an error in your browser. For example, in Google Chrome, the error looks like:

Error: "Received event for unconfigured/disabled/deleted index" but indexer acknowledgement is returning positives

If you see this error in messages or logs, edit your HEC token configurations to send data to an index that is able to accept data.

If indexer acknowledgment for your Amazon Kinesis Firehose data is successful but your data is not successfully indexed, the data may have been dropped by the parsing queue as an unparseable event. This is expected behavior when data is processed successfully in the input phase but cannot be parsed due to a logical error. For example, if the HTTP event collector is routing data to an index that has been deleted or disabled, the Splunk platform will still accept the data and begin processing it, which triggers indexer acknowledgement to confirm receipt. However, the parsing queue cannot pass the data to the index queue because the specified index is not available, thus the data does not appear in your index. For more information about the expected behavior of the indexer acknowledgement feature, see About HTTP Event Collector Indexer Acknowledgment.

If you suspect events have been dropped, search your "last chance" index, if you have one configured. If you are on Splunk Cloud, contact Splunk Support if you do not know the name of your last chance index. If you are on Splunk Enterprise, see the lastChanceIndex setting in indexes.conf for more information about the behavior of the last chance index feature and how to configure it.

Troubleshoot performance with the Splunk Monitoring Console

The Monitoring Console provides prebuilt dashboards with detailed topology and performance information about your Splunk Enterprise deployment.

• For Splunk Cloud Platform, see Introduction to the Cloud Monitoring Console.
• For Splunk Enterprise, see About the Monitoring Console.

Queue fill dashboard

If you are experiencing performance issues with your HEC server, you may need to increase the number of HEC-enabled indexers to which your events are sent.

Use the Monitoring Console to determine the queue fill pattern. Follow these steps to check whether your indexers are at capacity.

Steps

1. Navigate to either Monitoring Console > Indexing > Performance > Indexing Performance: Deployment or Monitoring Console > Indexing > Performance > Indexing Performance: Instance.
2. From the Median Fill Ratio of Data Processing Queues dashboard, select Indexing queue from the Queue dropdown and 90th percentile from the Aggregation dropdown.
3. (Optional) Set a Platform Alert to get a notification when one or more of your indexer queues reports a fill percentage of 90% or more. This alert can inform you of potential indexing latency.
• From Paid Splunk Cloud, navigate to Settings > Searches, reports, and alerts and select Monitoring Console in the app filter. Find the SIM Alert - Abnormal State of Indexer Processor platform alert, and click Edit > Enable to enable the alert.
• From the Splunk Enterprise Monitoring Console Overview page, click Triggered Alerts > Enable or Disable and then click the Enabled checkbox next to the SIM Alert - Abnormal State of Indexer Processor platform alert.

See determine queue fill pattern for an example of a healthy and unhealthy queue.

HTTP Event Collector dashboards

The Monitoring Console also comes with pre-built dashboards for monitoring the HTTP Event Collector. To interpret the HTTP event collector dashboards information panels correctly, be aware of the following:

• The Data Received and Indexed panel shows data as "indexed" even when the data is sent to a deleted or disabled index. Thus, this graph shows the data that is acknowledged by the indexer acknowledgment feature, even if that data is not successfully indexed. See the Error: 'Received event for unconfigured/disabled/deleted index' but indexer acknowledgment is returning positives section of this topic for more information about the expected behavior of the indexer acknowledgment feature when the index is not usable.
• The Errors panel is expected to show a steady stream of errors under normal operation. These errors occur because Amazon Kinesis Firehose sends empty test events to check that the authentication token is enabled, and the HTTP event collector cannot parse these empty events. Filter the Errors panel by Reason to help find significant errors.

For more information about the specific HTTP event collector dashboards, see HTTP Event Collector dashboards.

The HTTP event collector dashboards show all indexes, even if they are disabled or have been deleted.

Troubleshoot with prebuilt panels

The Splunk Add-on for Amazon Kinesis Firehose has four prebuilt panels that you can use to check if your data is being indexed for each index, indexer, or all indexers. There is also a panel that shows if your data is being properly load balanced across all indexers.

For information on how to use prebuilt panels, see Access prebuilt panels included with add-ons.