Understand entity status and search data in ITSI

Learn more about why an entity has an Unstable status in Splunk IT Service Intelligence (ITSI). Entities are considered active when searches continue finding data for the entity. Use the Entity Discovery Searches dashboard to view and configure the searches associated with a specific entity, troubleshoot issues with the entity status, and clean up inactive searches.

Access the Entity Discovery Searches dashboard

1. From ITSI, select Infrastructure Overview.
2. Find and select the entity you want to analyze. Alternatively, select the entity status under the Status column and you will be directly linked to the dashboard.
3. From the Entity Overview lister page, select the Entity Discovery Searches tab.

Investigate entity status using the Entity Discovery Searches dashboard

The Entity Discovery Searches dashboard displays all of the searches that are associated with the entity. These searches have discovered or are currently discovering data for the entity, and contributing to the status displayed for the entity. Use this information to take action on searches that need to be turned on or reconfigured, or confirm that your entities are displaying the correct status.

Dashboard panels

Panel	Description
Saved search title	The name of the saved search discovering the entity.
Last executed	Displays the last time that the search ran.
Entity status last marked active	The last time that the search discovered the entity.
Search active?	Displays if the search is turned on.
Tracks status?	Displays if the search is currently contributing to the entity's status.

Understand entity search details

1. Select a search from the list of searches displayed for the entity.
2. The dashboard displays the following details about each search:

   Title	Description
   Number of entities discovered	The number of entities in your environment that the search is discovering data for. This number may change depending on your configured search schedule.
   SPL Query	The query for the search.
   Cron schedule	Enter the schedule on which to run the search job.
   Next scheduled time	The next time that the search is configured to run.
   Earliest time	The earliest time for the time range of your search. For example, if earliest time is set to -90s and latest time is set to now, the search goes back 90 seconds.
   Latest time	The latest time for the time range of your search.
   Troubleshooting	A list of collapsible fields that describes the scenarios that are possibly causing an unstable or inactive entity status.

Troubleshoot unstable entities

The Entity Discovery Searches dashboard provides troubleshooting steps to fix searches that may be contributing to an unstable or inactive entity status. Use the information in these scenarios to troubleshoot your saved searches.

Data node isn't sending data

Cause
The host for the entity isn't sending data correctly.

Resolution
Confirm that the host is sending data, and check that the SPL search is properly configured on the Searches, Alerts, and Reports page. Run the search again to confirm that the search returns expected data.

Data ingestion and search look back time range not in sync

Cause
Data is being ingested at a slower rate than the search's configured cron schedule, or the look back time set by the earliest and latest time fields don't match the rate of data ingestion. For example, this issue would occur if data comes to the data index every hour, but the search is scheduled to run every 5 minutes with a look back time of 10 minutes.

Resolution
Update the cron schedule, earliest time, and latest time fields to match the data ingestion frequency on the Searches, Alerts and Reports page. Make sure the cron schedule and look back time overlap with the schedule of data ingestion. For example, the ITSI Import Objects - Perfmon saved search imports data for Windows entities. This search runs every minute with a look back time of 90 seconds. If you collect perfmon data every 5 minutes, with an average lag of 1 minute, update the search to run every 5 minutes and set the look back to 7 minutes in order to account for the delay.

Discovery search should not contribute to entity status

Cause
The search is incorrectly contributing to the entity's status calculation, and should not be tracking whether the entity is active.

Resolution

1. Select the Configure search button on the Entity Discovery Searches tab.
2. Update the action.itsi_import_objects.param.entity_status_tracking field for this search to 0 on the Searches, Alerts and Reports page. This turns off the search, and ensures this search will no longer contribute to the entity status calculation.

Searches are turned off or deleted

Cause
The search is turned off or deleted, and should not be contributing to the entity's status calculation.

Resolution
The entity status may be inaccurate if entities are linked to one or more searches that have been deleted or turned off. Even though these searches are no longer active, outdated search results can continue to exist in the entities that were linked to the search. To remove these searches, run the cleanupentitydiscoverysearches search command. For more information, see the following section about cleanupentitydiscoverysearches command.

Run a search command to clean up obsolete searches

You must have an admin role to run this command.

An entity's status may be inactive or disabled because the entity is associated with a search that is no longer discovering data, therefore causing an inaccurate entity status to display. The cleanupentitydiscoverysearches command removes disabled or deleted discovery searches from entities so that these searches no longer impact the entity status calculation. Run the cleanupentitydiscoverysearches command in the Search & Reporting app to find and remove these obsolete searches. See Search Summary view for more information.

1. Enter the search command in the search bar, for example:

   | cleanupentitydiscoverysearches

   
   You can also add these optional parameters to the search:

   Parameter	Description
   entity_batch_size	The number of entities that will be processed when the search runs. The default size is 1000.
   search_ids	A list of specific discovery search IDs. If you don't provide specific search IDs, any searches that have been recently deleted or turned off will be cleaned up from the entities. If you include search ID for an active search, it won't be removed.

   Here is an example search with optional parameters:

   | cleanupentitydiscoverysearches entity_batch_size=2000 search_ids="search1,search2"

   .
2. (Optional) Find logs for this search command by entering the search:

   index=_internal itsi_entity_discovery_search_cleaner

3. After the search runs, expand the search results to view the searches that have been cleaned up. Troubleshoot specific searches by using the value from the tid field in a separate search, for example:

   index=_internal <tid>

   .
4. (Optional) If you want this command to run on a regular schedule, select Settings > Searches, reports, and alerts and set a search schedule for the entity_discovery_search_cleaner search.