Troubleshoot missing findings in Splunk Enterprise Security

Issue

A detection is not generating findings as expected.

Causes

1. Findings are being suppressed.
2. The entire detection SPL search doesn't match, but part of it does.
3. The finding alert action isn't triggered.
4. Splunk Enterprise cannot parse the stash file.
5. The detection search schedule is incorrect, not running, or suppressed.
6. In a distributed architecture, you missed creating the notable index.

Solutions

1. Check to see if the notable index contains findings. Search in Splunk Web against the notable index to determine if the finding exists but is being excluded from the Analyst queue on the Mission Control page using the following search:

   index=notable

   Suppressions filter findings from appearing in the Analyst queue. If you see your finding in the index, then make sure that no suppressions are preventing the findings from appearing in the Analyst queue.

2. Run the detection manually over the given time frame and see if it matches the events. If it doesn't match, remove parts of the search until you isolate the part of the search that doesn't match.
3. Check the finding alert action logs. These logs indicate if the finding alert action is triggered to make a finding. Search in Splunk Web to view these logs:

   index=_internal sourcetype=notable_modalert

4. Verify that the search output doesn't include any unnecessary output. Make sure that the detection only outputs the fields you really need, and that the fields don't include extra content such as XML or excessive amounts of text. Extra content can make it difficult for Splunk to parse the stash file. If the stash file can't be parsed, then your findings might not be generated correctly.
5. Check the search scheduler logs. Search in Splunk Web to view the scheduler logs:

   index=_internal sourcetype=scheduler

   Look for the following:
   • Make sure that the search is running during the time-frame that you expect events
   • See if suppressed indicates that events are suppressed
   • See if result_count indicates that notable events are created, for example, is greater than one
   • Check the status field to make sure that the search is running successfully
6. If you are using a distributed architecture, create the notable index on your cluster.

See also

For more information on configuration settings and deploying indexes, see the product documentation:

• Configure findings manually to track specific fields in Splunk Enterprise Security
• Configure and deploy indexes in the Installation and Upgrade Manual