Troubleshoot the Splunk App for CEF

Issues after upgrade

See Upgrade an existing installation of the Splunk App for CEF for an explanation of implementation and deployment differences between the 1.0.0 and 2.0.0 versions, and guidance for planning to recreate the searches in the 2.0.0 app.

No data is arriving at receiver

If your syslog receiver is not receiving any data from Splunk Enterprise, follow these steps to troubleshoot.

1. Go to the Save Search step, and click the preview link to verify that your search is finding events. If it is not finding events, check the following items:
   • Verify that you can browse or search the data model directly to find events.
   • Verify that your data model is searching the correct indexes.
   • Verify that your data inputs are ingesting the data that you expect.
2. Search for index=_audit "action=search" and check the counts for the scan, event, and results for the searches you have created to ensure they match your expectations.
3. Confirm that you have deployed the Splunk Add-on for CEF Output to indexers. See Deploy the Splunk Add-on for CEF Output to indexers.
4. Verify that your receiver has the appropriate ports open and firewall rules in place to allow the data to arrive.

Selected data is not arriving at receiver

Determine if the receiver is the bottleneck

If you are not receiving the number of events that you expect, it is possible that the pace at which Splunk Enterprise is forwarding events is outpacing the rate at which the receiver is capable of ingesting them. If your output queues in Splunk Enterprise are full, this is the likely cause. You can reduce the number of indexers that you use for your CEF searches to slow performance to a speed that allows the receiver to ingest the events.

Work around batch indexed event problems

When you have a high volume of events that are indexed in batches, your events might have a significant gap between their _time and _indextime values, and the realtime dispatch might skip those events, so that they are not included in the data forwarded to your receiver.

If you observe this behavior, edit your CEF mapping searches in local/savedsearches.conf with one of the following workarounds.

Use a scheduled search with an appropriate offset

Edit your CEF mapping searches in local/savedsearches.conf to change the cron_schedule, dispatch.earliest_time and dispatch.latest_time.

For example, the following search is dispatched every 10 minutes, searches 10 minutes of data, and has a 10 minute offset.

[my search]
cron_schedule = */10 * * * *
dispatch.earliest_time = -20m@m
dispatch.latest_time = -10m@m

The resulting search can take up to double the offset time to reach your third-party syslog receiver.

Use a real-time search with an appropriate offset and disable backfill

Edit your CEF mapping searches in local/savedsearches.conf to change the dispatch.earliest_time, dispatch.latest_time, and dispatch.rt_backfill. Expect a very minimal delay in receiving events at the third-party receiver.

[my search]
dispatch.earliest_time = rt-10m@m
dispatch.latest_time = rt
dispatch.rt_backfill = false

Use a search that dispatches over indextime

This search looks for events that have been indexed in the prior 10 minutes regardless of the extracted time. To limit the number of buckets retrieved, allow for the extracted times to be up to 20 minutes in the past. Expect a very minimal delay in receiving events at the third-party receiver.

[my search]
cron_schedule = */10 * * * *
dispatch.earliest = -20m@m
dispatch.index_earliest = -10m@m
dispatch.index_latest = +0s
dispatch.latest = +0s

Requires selecting buckets based on X minutes, where X = 2 * offset.

Issues with TCP output and SSL groups

If you are not receiving data on the external system that was set during the package creation process, WARN level or ERROR level messages likely indicate that the output group isn't able to establish a connection. The following SPL search identifies issues with TCP output and SSL groups.

index=_internal (component=TcpOutputProc OR component=SSLCommon)

If your certificate password is wrong or the format is wrong, you will also see messages regarding SSL configuration errors.

Get more help

See Support and resource links for the Splunk App for CEF.