Troubleshoot your deployment

This topic describes how to troubleshoot Splunk Firehose Nozzle for VMware Tanzu.

You can't find your data

Are you searching for events and not finding them or looking at a dashboard and seeing "No result found"? Check Splunk Nozzle app logs. To view the nozzle's logs running on VMware Tanzu do the following: Log in as an admin via the CLI. Target the org created by the tile. View the recent app Splunk Nozzle logs (the version number installed by the tile will vary). Alternatively, you can stream the app logs as they're emitted. Here are a few common errors and possible resolutions:

Splunk configuration related errors

This error usually occurs when SSL is enabled on the Splunk HEC endpoint. Confirm that you're using https' in the Splunk HEC URL. This usually means the index value specified in the configuration doesn't exist on Splunk Host. Confirm that you're using the correct Splunk index value. This can occur when the Splunk HEC Token value is invalid. Confirm that you're using a valid token. This usually means that there was no valid SSL certificate found. Confirm that you're using a valid SSL certificate for the Splunk server, or set 'Skip SSL Validation' to true under Splunk settings. Note:Disabling SSL validation is not recommended for production environments. This error can occur when the Splunk server is offline or when the Splunk HEC URL is not valid. Confirm that both the Splunk server is running and that you're using a valid URL.

Cloud Foundry configuration related errors

This error can occur when the credentials provided for CF environment are invalid. Confirm that the API User and API Password each have access to the CF environment. This means that no valid SSL certificate was found. To remediate this error, provide a valid SSL certificate for Cloud Foundry or set 'Skip SSL Validation' to true under Cloud Foundry Settings.

Disabling SSL validation is not recommended for production environments.

The following troubleshooting tips assume you have access to Splunk to run basic searches against index _internal and the user-specified index for Firehose events.

Ensure Splunk Nozzle is forwarding events from the Firehose

Search app logs of the Nozzle to confirm correct behavior: A correct setup logs a start message with configuration parameters of the Nozzle logged as a JSON object, for example: Search app logs of the Nozzle for any errors: Errors are logged with corresponding message and stacktrace.

Check for dropped events due to HTTP Event Collector availability

As the Splunk Firehose Nozzle sends data to Splunk via HTTPS using the HTTP Event Collector, it is also susceptible to any network issues across the network path from point to point. Run the following search to determine if Splunk has indexed any events indicating issues with the HEC Endpoint.

Check for dropped events due to slow downstream(Network/Splunk)

If the nozzle emits the 'dropped events' warning saying that downstream is slow, then the network or Splunk environment might needs to be scaled. (eg. Splunk HEC receiver node, Splunk Indexer, LB etc) Run the following search to determine if Splunk has indexed any events indicating such issues.

Check for data loss inside the Splunk Firehose Nozzle

If "Event Tracing" is enabled, extra metadata will be attached to events. This allows searches to calculate the percentage of data loss inside the Splunk Firehose Nozzle, if applicable. Each instance of the Splunk Firehose Nozzle will run with a randomly generated UUID. The query below will display the message success rate for each UUID (Please update the index value based on your nozzle configuration).

Additional Resources

Checkout Troubleshoot HTTP Event Collector for troubleshooting issues related to HTTP Event Collector. Checkout Splunk Answers. Using the forum can save you time.