Create and manage search-driven lookups in Splunk Enterprise Security

A search-driven lookup lets you create a lookup based on the results of a search that runs at regular scheduled intervals. The search can run only against data stored in data models or in an existing lookup. Lookups created as search-driven lookups are excluded from bundle replication and are not sent to the indexers.

When to use search-driven lookups

Create a search-driven lookup if you want to know when something new happens in your environment, or need to consistently update a lookup based on changing information from a data model or another lookup.

The search-driven lookup collects and stores information from data models or other lookups. The data stored in the lookup represents a historical summary of selected fields gathered from events. You can view changes on a dashboard or use a correlation search to compare data from the search-driven lookup with new events, and alert if there is a match. For example, to find out when a new user logs in to a web server.

1. Search for user data in the Authentication data model and filter by the web server host name with the where command.
2. Verify the search results match the known hosts and users in your environment.
3. Create a guided search-driven lookup to collect and store information on a recurring schedule about users logging in to the web servers.
4. Create a correlation search that alerts you when a user logs in to one of the web servers that he or she has not accessed in the past, based on the historical information in the search-driven lookup.

Create a search-driven lookup

When you create a search-driven lookup, two knowledge objects are created. One knowledge object is the lookup that is generated by the search, while the other knowledge object is the search that drives the lookup.

Create a search-driven lookup as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Content> Content Management.
2. Click Create New Content and select Search-Driven Lookup.
3. (Optional) Select an App. The default app is SplunkEnterpriseSecuritySuite. You can create the lookup in a specific app, such as SA-NetworkProtection, or a custom app. You cannot change the app after you save the search-driven lookup.
4. (Optional) Type a description for the search.
5. Type a label for the lookup. This is the name of the search-driven lookup that appears on Content Management.
6. Type a name for the lookup. After you save the lookup, the name cannot be changed.
7. Type a cron schedule to define how often you want the search to run.
8. Select real-time or continuous scheduling for the search. Real-time scheduling prioritizes search performance, while continuous scheduling prioritizes data integrity.
9. Type a Search Name to define the name of the saved search. After you save the lookup, the name cannot be changed.
10. Select a mode of Guided to create a search without having to write the search syntax yourself, or select Manual to write your own search. See the example for help building a search with the guided search editor.
11. If you create a search in manual mode, type a search.
12. (Optional) Use the Enabled toggle to enable retention.
    1. In the Time field list, type a valid time field for retention. Note that this is a free-form text field, and there is no validation on this field.
    2. In the Earliest Time field, type the time specifier such as -1y to retain data for one year.
    3. In the Time Format field, type the time format such as %s for seconds.
       See Date and time format variables in the Splunk Enterprise Search Reference.
13. Click Save to save the search.

Example search-driven lookup

In this example search-driven lookup included with Splunk Enterprise Security, you want to track attacks identified by your intrusion detection system (IDS). You can then be notified of new attacks with a correlation search, or determine whether an attack is new to your environment or not. The Intrusion Center dashboard uses this search-driven lookup for the New Attacks - Last 30 Days panel. See Intrusion Center dashboard.

1. From the Splunk Enterprise Security menu bar, select Configure > Content > Content Management.
2. Click Create New Content and select Search-Driven Lookup.
3. (Optional) Select an App of SA-NetworkProtection. You cannot change the app after you save the search-driven lookup.
4. Type a description of "Maintains a list of attacks identified by an IDS and the first and last time that the attacks were seen."
5. Type a label of IDS Attack Tracker Example for the lookup. This is the name of the search-driven lookup that appears on Content Management.
6. Type a unique and descriptive name for the lookup of ids_attack_tracker_example. After you save the lookup, the name cannot be changed.
7. Type a cron schedule to define how often you want the search to run. If your IDS collects data often, type a cron schedule of 25 * * * * to run the search at 25 minutes every hour every day.
8. Select a Continuous Schedule because the lookup must track all data points.
9. Type a Search Name of Network - IDS Attack Tracker - Example Lookup Gen.
10. Select guided mode to use the guided search editor to create the search.
11. Click Open guided search editor to start creating the search.
12. Select a data source of Data Model because the IDS Attack data is stored in a data model.
13. Select a data model of Intrusion_Detection and a data model dataset of IDS_Attacks.
14. Select Yes for the summaries only field to run the search against only the data in the accelerated data model.
15. Select a time range that uses Relative time that begins with an earliest time of 70 minutes ago, starting at the beginning of the minute, and ends now. Click Apply to save the time range.
16. Click Next.
17. (Optional) Type a where clause to filter the data from the data model to only the data from a specific IDS vendor and click Next.
18. Add aggregate values to track specific statistics about the data and store that information in the lookup. At least one aggregate is required.
    1. To track the first time that an IDS attack was seen in your environment, add a new aggregate with a function of min and a field of _time and save it as firstTime.
    2. Track the last time an attack was seen by adding another aggregate with a max function and a field of _time and saving it as lastTime. This creates two columns in the lookup, firstTime and lastTime.
19. Add split-by clauses to track more data points in the lookup. All split-by clauses appear as columns in the lookup.
    1. Add a split-by clause of IDS_Attacks.ids_type and rename it as ids_type to monitor the IDS type in the lookup.
    2. Add a split-by clause to rename IDS_Attacks.signature as signature.
    3. Add a split-by clause to rename IDS_Attacks.vendor_product as vendor_product.
20. Click Next.
21. Select a retention period that defines the age of the data to be stored in the lookup. For example, you want to keep 5 years of IDS attack evidence stored in this lookup. Select a time field of lastTime to base the retention on the last time an attack was identified by the IDS. Type an earliest time of -5y and indicate the format of the time value that you entered: %s. You can find guidance on the time format in the Splunk platform documentation.
• For Splunk Enterprise, see Date and time format variables in the Splunk Enterprise Search Reference manual.
• For Splunk Cloud Platform, see Date and time format variables in the Splunk Cloud Platform Search Reference manual.




22. Click Next.
23. Review the search created by the wizard and click Done to finish using the guided search editor.
24. Click Save to save the search.

Modify a search-driven lookup

Since a search-driven lookup contains the two knowledge objects of search and lookup, there are two ways to modify it. Both ways will open the search-driven lookup editor.

Modify the search-driven lookup as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Content > Content Management.
2. Select a Type of Search-Driven Lookup.
3. Click the lookup that you want to edit.
4. Make changes and click Save.

Modify the lookup generating search as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Content > Content Management.
2. Select a Type of Lookup Generating Search.
3. Click the lookup that you want to edit.
4. Make changes and click Save.

Modify retention settings for a search-driven lookup

You can modify search-driven lookup retention settings for performance purposes.

As of Enterprise Security 6.3.0, retention settings are no longer handled in the custom search builder specification of the savedsearches.conf file. The search-driven lookup retention is managed by the lookup_retention.py modular input using managed_configurations settings. Therefore, you no longer use the guided search builder to revise the retention settings in the search processing language (SPL). With retention settings migrated into managed_configurations, the retention is no longer impacted if you use outputlookup append=T in the SPL of a search driven lookup, so the change delta does not get ignored. In addition, for CSV only, the outputlookup override_if_empty is set to true by default and allows an outputlookup to delete the output file if the result set is empty. If you have existing retention settings, they remain as you set them.

Modify the retention settings as follows:

1. From the Splunk Enterprise Security menu bar, select Configure > Content > Content Management.
2. Select a Type of Search-Driven Lookup.
3. Click the lookup that you want to edit.
4. Scroll to Retention.
5. If disabled, use the Enabled toggle to enable retention.
6. In the Time field list, type a valid time field for retention.
   Time fields are defined in the transforms.conf file. Examples include the following:
   • _time
   • lastTime
7. In the Earliest Time field, type the time specifier such as -1y to retain data for one year.
8. In the Time Format field, type the time format such as %s for seconds.
   For Splunk Enterprise, see Date and time format variables in the Splunk Enterprise Search Reference manual.
   For Splunk Cloud Platform, see Date and time format variables in the Splunk Cloud Platform Search Reference manual.

The default search-driven lookup retention settings are as follows. Those listed as N/A are not available for modifying through the Splunk Web UI.

Search Driven Lookup Label	Search Driven Lookup Description	Time Field	Retention Period
Access App Tracker	Maintains a list of Authentication app values and the first and last time they have been seen.	_time	5 years
Access Tracker	Maintains a list of users that have authenticated to each system and the first, second to last, and last time they have been seen.	lastTime	1 year
Asset/Identity Categories	Maintains a list of categories that apply to assets and identities.	N/A	N/A
Correlation Searches Lookup	Maintains correlation search enrichment for notable events.	N/A	N/A
ES Notable Events	Maintains a list containing pertinent information for the last 48 hours of notable events.	N/A	N/A
Firewall Rule Tracker	Maintains a list of Traffic rule values by device and vendor and the first and last time they were seen. See Firewall Rule Tracker Retention	year	2 years
IDS Attack Tracker	Maintains a list of IDS attacks by vendor and the first and last time they were seen.	lastTime	5 years
IDS Category Tracker	Maintains a list of IDS attack categories by vendor and the first and last time they were seen.	lastTime	5 years
Licensing - Events Per Day	Maintains a list of event counts per day per index.	_time	1 year
Listening Ports Tracker	Maintains a list of all port and protocol combinations listening on each system and the first and last time they were seen.	lastTime	5 years
Local Processes Tracker	Maintains a list of all processes on each system and the first and last time they were seen.	lastTime	1 month
Malware Operation Tracker	Maintains a list of anti-malware product and signature versions for each system.	_time	1 year
Malware Tracker	Maintains a list of all detections (regardless of status) for each system and the first and last time they were seen.	lastTime	5 years
PCI Domain Lookup	Maintains a list of pci domains that apply to assets and identities.	N/A	N/A
Port/Protocol Tracker	Maintains a list of allowed Traffic by unique transport protocol and destination port combination and the first and last time they were seen.	lastTime	5 years
Registry Tracker	Maintains a list of registry paths, keys, and value information by system and the first and last time they were seen.	lastTime	1 year
Services Tracker	Maintains a list of all services (and the most recent startmode) for each system and the first and last time they were seen.	lastTime	1 month
System Version Tracker	Maintains a list of the most recent operating system version for each system and the time we got this information.	_time	5 years
Traffic Bytes Tracker	Maintains Traffic byte statistics.	N/A	N/A
Update Signature Reference	Maintains a list of all updates by vendor and the first and last time they were seen.	lastTime	1 year
URL Length Tracker	Maintains Web user agent length statistics.	N/A	N/A
User Accounts Tracker	Maintains a list of all local user accounts on each system and the first and last time they were seen (not accelerated).	lastTime	1 year
User Agent Length Tracker	Maintains Web url length statistics.	N/A	N/A
Vulnerability Signature Reference	Maintains a list of vulnerability signatures by vendor (including external reference information such as cve) and the first and last time they were seen.	lastTime	1 year
Vulnerability Tracker	Maintains a list of Vulnerabilities by signature, destination and the first and last time they were seen.	lastTime	5 years
Whois Tracker	Maintains a list of whois scan data including the resolved_domain (if domain was an IP) and the date the domain was created.	_time	5 years

Global settings for search-driven lookup retention is handled by the data_retention_manager in Settings > Data Inputs > Lookup Retention.

Firewall rule tracker retention

The Firewall Rule Tracker retention works differently from the others. It uses only the year field in its retention spec, which means that a relative time is used that's based off the beginning of the year. The default retention period is set to two years, in order to preserve data quantity. For example, if today is 06/20/2020 and your retention period is "-1y", then all rows in your lookup where the year is less than or equal to 2019 are deleted.

Do not set the Firewall Rule Tracker retention period to less than two years, unless you accept the possibility of data loss.

Enable or disable the search populating a search-driven lookup

You can enable or disable the search of a search-driven lookup to prevent the search from updating the lookup. If you disable the search that populates a search-driven lookup, the search stops updating the lookup and the data in the lookup will stop being updated. Correlation searches or dashboards that rely on the data inside the lookup will be out-of-date.

1. Select Configure > Content > Content Management.
2. Filter on a type of search-driven lookup and open the search-driven lookup that you want to enable or disable.
3. Find the Search name of the search-driven lookup.
4. From the Splunk platform menu bar, select Settings > Searches, reports, alerts.
5. (Optional) Filter by Type and App of All.
6. Find the search and enable or disable it.