Splunk® Mission Control

Get Data into Splunk Mission Control

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Troubleshoot why data is not getting into Splunk Mission Control from Splunk Enterprise Security

If data that you get into Splunk Mission Control from Splunk Enterprise Security fails to appear in Splunk Mission Control, follow these troubleshooting steps.

If you are a Splunk Cloud Platform customer, contact Splunk Customer Support for assistance troubleshooting why data is not getting into Splunk Mission Control from Splunk Enterprise Security.

All of these troubleshooting steps assume that you have successfully done the following:

  1. Verified that your deployment meets the installation requirements for Splunk Connect for Mission Control. See Installation requirements for Splunk Connect for Mission Control.
  2. Installed Splunk Connect for Mission Control. See Install Splunk Connect for Mission Control.
  3. Set up certificates to secure getting data into Splunk Mission Control from Splunk Enterprise Security. See Set up certificates to secure getting data into Splunk Mission Control from Splunk Enterprise Security.
  4. Set up Splunk Connect for Mission Control to get data into Splunk Mission Control from Splunk Enterprise Security. See Set up getting data into Splunk Mission Control from Splunk Enterprise Security.

For a diagram and an overview of how Splunk Connect for Mission Control gets data into Splunk Mission Control from Splunk Enterprise Security, see How Splunk Connect for Mission Control gets data into Splunk Mission Control from Splunk Enterprise Security.

Notable events do not appear on the analyst queue in Splunk Mission Control

If you do not see any notable events on the analyst queue in Splunk Mission Control, several reasons could be the root cause.

Cause: filters or permissions on the analyst queue might be preventing you from seeing data

If you do not see any notables in Splunk Mission Control, filters and permissions could prevent you from seeing data.

  1. In Splunk Mission Control, click Investigations to open the analyst queue.
  2. Review the filter settings on the page and in the URL.
    If your view is filtered, click Clear filters.
  3. If you still do not see any notables, check the permissions of your user role. See Manage access and permissions for users of Splunk Mission Control in Set Up and Customize Splunk Mission Control.

Cause: there might not be any notable events to send from Splunk Enterprise Security

If there aren't any new notable events being generated in Splunk Enterprise Security, you might not see any notables in Splunk Mission Control. To troubleshoot, confirm that data exists to send to Splunk Mission Control.

  1. From the Splunk Enterprise Security search head with Splunk Connect for Mission Control installed, click Search > Search to open the search page.
  2. For a time range of the last 15 minutes, search for notable events.

    `notable`

  3. If that search returns no results, expand the time range.
  4. If that search still returns no results, make sure that correlation searches are running and creating notable events. Run the following search to see the list of enabled correlation searches that create notable events.

    | rest splunk_server=local count=0 /servicesNS/-/SplunkEnterpriseSecuritySuite/saved/searches | where match('action.correlationsearch.enabled', "1|[Tt]|[Tt][Rr][Uu][Ee]") | where disabled=0 | eval actions=split(actions, ",") |search actions=notable | table title,actions

  5. If that search returns no results, enable correlation searches. See Configure correlation searches in Splunk Enterprise Security in the Administer Splunk Enterprise Security manual.

Cause: search to locate and send notable events is not running or fails to return results

Confirm that notable events are being sent successfully from Splunk Enterprise Security and Splunk Connect for Mission Control.

  1. Make sure that the search to forward notable events is enabled and running. From the Splunk Enterprise Security search head with Splunk Connect for Mission Control installed, select Settings > Searches, reports, and alerts.
  2. Change the App filter to Splunk Connect for Mission Control.
  3. Locate the Mission Control - Forward Notable Events and determine if it is enabled.
    1. If the search is disabled, enable it.
    2. If the search is enabled, confirm that there is a Next Scheduled Time listed. If there is no Next Scheduled Time listed, click Edit > Edit Alert to confirm that the search is scheduled.
    3. If the search is enabled and scheduled, run the search manually. Click Run to run the search in another tab.
  4. Confirm that the search runs without errors and returns results.
  5. If the search returns errors, fix the search syntax so that the search runs correctly.
  6. If the search returns no results, there are no notable events in the time range to send. You might consider expanding the time range for the alert or waiting for new notable events to appear.

Cause: notable events are stuck waiting to be sent

Splunk Connect for Mission Control uses a search to retry sending notable events that were not successfully sent with the saved search used to send notable events. After five attempts to retry sending notable events, the search stops trying to send the notable events. If you see notable events that are stuck waiting to be sent for more than half an hour, they might have reached the maximum number of retry attempts and are stuck. Follow these troubleshooting steps to determine if that is the case and to forward those events.

  1. From the Splunk Enterprise Security search head with Splunk Connect for Mission Control installed, click Audit > Event Ingestion Audit to open the Event Ingestion Audit dashboard.
  2. Review the dashboard to see if there are events waiting to be sent. If events are still waiting to be sent after 10 minutes, continue troubleshooting.
  3. Confirm that the search to retry unacknowledged events is enabled and running. Select Settings > Searches, reports, and alerts.
  4. Change the App filter to Splunk Connect for Mission Control.
  5. Locate the Mission Control - Retry Unacknowledged Events and determine if it is enabled.
    1. If the search is disabled, enable it.
    2. If the search is enabled, confirm that there is a Next Scheduled Time listed. If there is no Next Scheduled Time listed, click Edit > Edit Alert to confirm that the search is scheduled.
    3. If the search is enabled and scheduled, run the search manually. Click Run to run the search in another tab.
  6. Check the status of the event acknowledgement for the notable events by reviewing the contents of the event acknowledgement lookup. Run the following search:

    | inputlookup event_ack_lookup | where ack_received=0

    1. If the retries count is less than five, the events might still be sent by the retry search. Events are sent every 5 minutes by that search.
    2. If the number of retries is set to five, the retry search does not try to send them again. If you want the retry search to try to forward them, update the search.
  7. Update the maximum number of retries in the retry unacknowledged events saved search. Select Settings > Searches, reports, and alerts.
  8. Change the App filter to Splunk Connect for Mission Control.
  9. Locate the Mission Control - Retry Unacknowledged Events and select Edit > Edit Alert.
  10. Increase the number of retry attempts from 5 to 10. Modify the subsearch at the end of the search from:

    ... | search [inputlookup event_ack_lookup | where ack_received=0 AND _time<relative_time(now(), "-5m") AND retries<5 | table event_id | head 25]

    to

    ...|search [inputlookup event_ack_lookup | where ack_received=0 AND _time<relative_time(now(), "-5m") AND retries<10 | table event_id | head 25]

  11. Wait for the search to run again and confirm that it sends the additional notable events.

Cause: notable events are skipped and not sent to Splunk Mission Control

If you see notable events in Incident Review in Splunk Enterprise Security, but they do not exist in Splunk Mission Control and are not waiting to be sent to Splunk Mission Control, the events have been skipped. This can happen if there is a several minute difference between the _time and the index_time values for the notable event.

  1. Identify a notable event that exists in Splunk Enterprise Security but is missing from Splunk Mission Control.
    1. In Splunk Enterprise Security on the Incident Review page, locate the notable event that was not sent to Splunk Mission Control.
    2. Expand the notable event row and copy the event_id for the notable event.
  2. Determine if the notable event was sent to Splunk Mission Control by looking for it in the event acknowledgement lookup. Run the following search:

    |inputlookup event_ack_lookup2 | search event_id=<event_id_you_copied>

    1. If no results are returned, continue troubleshooting.
    2. If results are returned, review the results to determine if something else is preventing the notable event from appearing in Splunk Mission Control, such as the maximum number of retries for sending the event have been reached.
  3. Determine if the notable event has different values for _time and _indextime fields. Run the following search for the time range that includes the missing notable event:

    `notable` | eval indextime = strftime(_indextime, "%FT%T") | table _time, indextime

    1. If the _time and _indextime values are more than 2 minutes apart, update the search that sends notable events.
    2. If the _time and _indextime values are not substantially different, something else could be causing the issue.
  4. Update the search that sends notable events to prevent notable events from being skipped in the future.
    1. From the Splunk Enterprise Security search head with Splunk Connect for Mission Control installed, select Settings > Searches, reports, and alerts.
    2. Change the App filter to Splunk Connect for Mission Control.
    3. Locate the Mission Control - Forward Notable Events search and click Edit > Edit Alert.
    4. Modify the search syntax to match the following:

      `notable` | where _indextime>relative_time(now(), "-2m@m") | join [rest splunk_server=local /services/server/info | fields guid | rename guid as source_instance_id] | join [rest splunk_server=local /servicesNS/nobody/splunk-connect-for-mission-control/configs/conf-proxy/product_management | fields deployment_id | rename deployment_id as proxy_id] | resolveartifacts

    5. Modify the time range of the search to Last 60 minutes.
    6. Click Save.
  5. To send the notable events that were already skipped to Splunk Mission Control, run an ad hoc search to forward the notable events. Identify the time range of the notable events that were not sent, or the time range of past notable events that you want to send to Splunk Mission Control. For that time range, run the following search:

    `notable` | join [rest splunk_server=local /services/server/info | fields guid | rename guid as source_instance_id] | join [rest splunk_server=local /servicesNS/nobody/splunk-connect-for-mission-control/configs/conf-proxy/product_management | fields deployment_id | rename deployment_id as proxy_id] | resolveartifacts | sendalert send2mc_notable

  6. Verify that the notable events make it to Splunk Mission Control.

Cause: a delay is happening with Splunk Mission Control

If a delay in Splunk Mission Control causes notable events to not be successfully processed, you can tell from your Splunk Enterprise Security search head with Splunk Connect for Mission Control installed.

  1. From your search head, open Splunk Connect for Mission Control.
  2. Click Audit > Event Ingestion Audit.
  3. Review the Events Waiting To Be Sent To Mission Control panel. If there are more than a hundred events waiting to be sent, you can temporarily change the number of events sent from 25 to something higher.
  4. Select Settings > Searches, reports and alerts.
  5. Change the App filter to Splunk Connect for Mission Control.
  6. Locate the Mission Control - Retry Unacknowledged Events
  7. Click Edit > Edit Alert.
  8. Update the search syntax to include a larger number after the head command. For example,

    ... | head 100

  9. Save the changes.


Cause: you are sending more than 1000 notables per hour total to Splunk Mission Control

Splunk Mission Control limits the total number of notables that can be sent to each tenant of Splunk Mission Control to 1000 notables per hour. If you send more than 1000 notables, additional notables are dropped and you receive a 429 HTTP status code in response. You can limit the number of notables being sent to Splunk Mission Control in several ways:


Artifacts are missing for notable events sent from Splunk Enterprise Security

If you don't see any or all of the artifacts you expect to see on the Analytics tab when investigating a notable in Splunk Mission Control, follow these troubleshooting steps.

Cause: Splunk Connect for Mission Control is missing the artifacts lookup file

If Splunk Connect for Mission Control is unable to use the included lookup table file for artifact field identification, cim_artifact_fields.csv, artifacts do not get extracted from the notable events forwarded to Splunk Mission Control.

  1. From the Splunk Enterprise Security search head with Splunk Connect for Mission Control installed, open the command line.
  2. Open the lookups directory of Splunk Connect for Mission Control.
    cd /$SPLUNK_HOME$/etc/apps/splunk-connect-for-mission-control/lookups
  3. Print the contents of the directory.
    ls
    1. If the lookup files are listed as cim_artifact_fields.csv.default and custom_artifact_fields.csv.default, type the following commands to correct the file name:
      mv cim_artifact_fields.csv.default cim_artifact_fields.csv
      mv custom_artifact_fields.csv.default custom_artifact_fields.csv.
    2. If the lookup files are listed as cim_artifact_fields.csv and custom_artifact_fields.csv, continue troubleshooting.

If this was the root cause of artifacts not being sent, it's likely that Splunk Connect for Mission Control was incorrectly installed, or the mc_lookup_handler script that runs after installation failed to run. The script runs on every search head after installation to make sure that the .default extension is removed from the lookup files included with Splunk Connect for Mission Control.

Cause: the search to forward notable events is missing a required search command

If you modified the search to forward notable events to Splunk Mission Control and the search no longer includes the resolveartifacts search command, you must update the search.

  1. From the Splunk Enterprise Security search head with Splunk Connect for Mission Control installed, select Settings > Searches, reports, and alerts.
  2. Change the App filter to Splunk Connect for Mission Control.
  3. Locate the Mission Control - Forward Notable Events saved search.
  4. Click Edit > Edit Alert.
  5. Review the search syntax and determine if the search includes or ends with |resolveartifacts.
    If it does not, update the search to add it:

    ... | resolveartifacts

  6. If the search includes the required custom search command, continue troubleshooting.

See Customize getting data into Splunk Mission Control from Splunk Enterprise Security for guidance in safely modifying the search to forward notable events to Splunk Mission Control.

Cause: the missing artifact is a custom field not included in a CIM data model

If you see some artifacts listed on the Analytics tab for a notable for Splunk Mission Control, but not all of the ones that you expect to see, you might need to customize the search used to send notable events to Splunk Mission Control. See Customize getting data into Splunk Mission Control from Splunk Enterprise Security, specifically the section called Extract custom fields as artifacts in Splunk Mission Control.

Cause: the alert action to send risk events does not work

If you expect to see risk scores in Splunk Mission Control from your Splunk Enterprise Security deployment for the artifacts in the analytics tab, but none appear, follow these troubleshooting steps.

  1. From the Splunk Enterprise Security Instance with Splunk Connect for Mission Control installed, select Settings > Searches, reports, and alerts.
  2. Change the App: filter to Splunk Connect for Mission Control.
  3. Change the Owner filter to All.
  4. Locate the saved search Mission Control - Forward Notable Events - Ingestion and click Edit > Clone.
    1. In the cloned alert, update the title to Mission Control - Forward Notable Events.
    2. In Permissions click Clone. All other settings can remain as is.
    3. Click Clone Alert.
  5. Click the new saved search that appears, Mission Control - Forward Notable Events and click Enable.
  6. Locate the previous saved search Mission Control - Forward Notable Events - Ingestion and click Edit > Disable.

You might need to adjust the Owner filter to view the new saved search based on what user created it. When you ingest new notables with risk scores into Splunk Mission Control, the risk scores appear.

Content Management data is not visible in Splunk Mission Control

If you expect to see knowledge objects in Splunk Mission Control from your Splunk Enterprise Security deployment, but none appear, follow these troubleshooting steps.

Cause: the search to forward content management data is not enabled

  1. From the Splunk Enterprise Security search head with Splunk Connect for Mission Control installed, select Settings > Searches, reports, and alerts.
  2. Change the App filter to Splunk Connect for Mission Control.
  3. Locate the Mission Control - Forward Content Management Data search and enable it.
  4. If the search is already enabled, contact Splunk Customer Support.
Last modified on 27 August, 2021
PREVIOUS
Troubleshoot setting up Splunk Connect for Mission Control
  NEXT
Troubleshoot updating notable status information in Splunk Enterprise Security

This documentation applies to the following versions of Splunk® Mission Control: Current


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters