Specify the time to run detections in Splunk Enterprise Security
Set the time to run detections as scheduled searches to specify the results that get included in the search results. Splunk Enterprise Security offers the following timestamp options to find the events that get included in the detections:
- Event time
- Index time
By default, detections use Event Time , also known as search time or extracted time, which corresponds to the time when events are logged. However, using Event Time can result in detections disregarding delayed events.
Index time is the time when Splunk Enterprise Security indexes events, which is the time when a Splunk indexer receives an event. Searching on Index time as soon as the event is indexed, might generate an alert.
Selecting Index Time when configuring detections circumvents event lag and improves security alerting. This is because configuring detections using the Index time range can more effectively monitor data that arrives late and run the detections against that data. Therefore, configure and run detections by Index time to avoid the time lag in the search results and focus on the most recent events during an investigation.
For example: Deploy a detection (R1) that runs every five minutes and checks for a particular scenario (S1) within the last 5 minutes to fire an alert whenever S1 is found. Detections are based on extracted time. So, when S1 events are delayed by five minutes, no alerts might be triggered by R1 because the 5-minute window checked by the continuous, scheduled R1 never re-scans the events from a previous, already checked window. Despite those delayed events being known to exist, R1 is already set to check another time window, thereby, missing the opportunity to detect S1 behavior from delayed events. When detections use extracted time, some events may land on the indexers a bit later due to a transport bottleneck such as network or processing queue. Event lag is a common concern for some Cloud data sources because some data might take an extended period of time to come into Splunk after the event is generated.
Limitations using index time
Following are some limitations when using Index time range in a detection:
- Selecting Index time as the time range for a detection might impact the performance of the detection.
- Select Index time to run a detection only on raw events that do not use accelerated data model fields or the
tstats
command in the search. Otherwise, the UI might display errors. You can update the detection so that it does not include anytstats
commands to avoid these errors. - Drill down searches for findings might get modified when using Index time.
- Index time filters are added after the first " | " pipe character in a search string. Index time filters do not have any effect on accelerated data models, sub-searches,
stats
, streaming, orlookup
commands. So, custom drill-down searches must be constructed correctly when using Index time. - Index time might not apply correctly to the Contributing Findings search for risk findings.
Configure the time range in the detection editor
Follow these steps to configure the time range in the detection editor to run detections:
Using '''Index time''' might impact the logic analytics of the search. Detections typically expect related events to be in the data set around the same time. However, this might not be the case when searching by '''Index time'''.
- In Splunk Enterprise Security, select Configure.
- Select Content management.
- In the Content management page, locate the detection that you want to configure to use index time.
This opens the Edit finding-based detection editor. - Go to Time range.
- In Time range, select the checkbox "Index time" to use the index time for the correlation search results.
Alternatively, select the checkbox Event time to use the extracted time for the correlation search results.When you select '''Index time''' to run the detection, all the underlying searches run using the '''All Time''' time range picker, which might impact the search performance. This includes the detection as well as the drill-down search of the finding adaptive response action. Additionally, the drill down search for the finding in the '''Mission Control''' page also uses index time.
- Modify the Earliest time and Latest time time modifier to specify custom and relative time ranges. You can specify an exact time such as earliest="10/5/2021:20:00:00", or a relative time such as earliest=-h or latest=@w6.
- Modify the Cron schedule to control how frequently the search runs.
- Select one of the two Scheduling options: Real time or Continuous.
Detections can run with a real-time or continuous schedule. Use a real-time schedule to prioritize current data and performance. Detections with a real-time schedule are skipped if the search cannot be run at the scheduled time. Detections with a real-time schedule do not backfill gaps in data that occur if the detection is skipped. Use a continuous schedule to prioritize data completion, as detections with a continuous schedule are never skipped. - Specify a Schedule window for the detection. Type 0 to not use a schedule window. Type auto to use the automatic schedule window set by the scheduler. Type a number that corresponds with the number of minutes that you want the schedule window to last. When there are many scheduled reports set to run at the same time, specify a schedule window to allow the search scheduler to delay running this search in favor of higher-priority searches.
- Specify a Schedule priority for the search.
Change the default to Higher or Highest depending on how important it is that this search runs, and that it runs at a specific time. The schedule priority setting overrides the schedule window setting, so you do not need to set both.If you manually convert a real-time search to a scheduled search, this does not automatically adjust the earliest or latest dispatch times. The time range default remains the same as the original real-time search, such as -5m@m ~ +5m@m which does discard events based on the extracted time being slightly in the future versus in the past. You will also need to evaluate the syntax of the converted search. This is because
| data model
is in use for real-time searches. However, if you are moving to a scheduled search, you can use the| tstats
command for efficiency. - Specify a Max append time for the search to indicate the maximum time range to group findings and intermedaite findings. For example, 7d. The default value is 30d.
See also
For more information on time ranges and scheduling detections to run in Splunk Enterprise Security, see the product documentation:
- Troubleshooting event indexing delay in the Splunk Enterprise Troubleshooting Manual
- Index time versus search time in the Splunk Enterprise Managing Indexers and Clusters of Indexers Manual
- Time modifiers in the Splunk Cloud Services SPL2 Search Manual
- Turn on detections in Splunk Enterprise Security
Add annotations to detections in Splunk Enterprise Security | Turn on detections in Splunk Enterprise Security |
This documentation applies to the following versions of Splunk® Enterprise Security: 8.0.0
Feedback submitted, thanks!