Manage summary index gaps
The accuracy of your summary index searches can be compromised if the summary indexes involved have gaps in their collected data.
Gaps in summary index data can come about for a number of reasons:
- A summary index initially only contains events from the point that you start data collection: Don't lose sight of the fact that summary indexes won't have data from before the summary index collection start date--unless you arrange to put it in there yourself with the backfill script.
- Splunk deployment outages: If your Splunk deployment goes down for a significant amount of time, there's a good chance you'll get gaps in your summary index data, depending on when the searches that populate the index are scheduled to run.
- Searches that run longer than their scheduled intervals: If the search you're using to populate the summary index runs longer than the interval that you've scheduled it to run on, then you're likely to end up with gaps because Splunk software won't run a scheduled search again when a preceding search is still running. For example, if you were to schedule the index-populating search to run every five minutes, you'll have a gap in the index data collection if the search ever takes more than five minutes to run.
Note: For general information about creating and maintaining summary indexes, see "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.
Use the backfill script to add other data or fill summary index gaps
If you have Splunk Enterprise, you can use the
fill_summary_index.py script, which backfills gaps in summary index collection by running the saved searches that populate the summary index as they would have been executed at their regularly scheduled times for a given time range. In other words, even though your new summary index only started collecting data at the start of this week, if necessary you can use
fill_summary_index.py to fill the summary index with data from the past month.
In addition, when you run
fill_summary_index.py you can specify an App and schedule backfill actions for a list of summary index searches associated with that App, or simply choose to backfill all saved searches associated with the App.
When you enter the
fill_summary_index.py commands through the CLI, you must provide the backfill time range by indicating an "earliest time" and "latest time" for the backfill operation. You can indicate the precise times either by using relative time identifiers (such as
-3d@d for "3 days ago at midnight") or by using UTC epoch numbers. The script automatically computes the times during this range when the summary index search would have been run.
NOTE: To ensure that the
fill_summary_index.py script only executes summary index searches at times that correspond to missing data, you must use
-dedup true when you invoke it.
fill_summary_index.py script requires that you provide necessary authentication (username and password). If you know the valid Splunk Enterprise key when you invoke the script, you can pass it in via the
The script is designed to prompt you for any required information that you fail to provide in the command line, including the names of the summary index searches, the authentication information, and the time range.
Examples of fill_summary_index.py invocation
If this is your situation:
You need to backfill all of the summary index searches for the splunkdotcom App for the past month--but you also need to skip any searches that already have data in the summary index:
Then you'd enter this into the CLI:
./splunk cmd python fill_summary_index.py -app splunkdotcom -name "*" -et -mon@mon -lt @mon -dedup true -auth admin:changeme
If this is your situation:
You need to backfill the
my_daily_search summary index search for the past year, running no more than 8 concurrent searches at any given time (to reduce impact on performance while the system collects the backfill data). You do not want the script to skip searches that already have data in the summary index. The
my_daily_search summary index search is owned by the "admin" role.
Then you'd enter this into the CLI:
./splunk cmd python fill_summary_index.py -app search -name my_daily_search -et -y -lt now -j 8 -owner admin -auth admin:changeme
Note: You need to specify the
-owner option for searches that are owned by a specific user or role.
What to do if fill_summary_index.py is interrupted while running
fill_summary_index.py is interrupted, look for a
log directory in the app that you are invoking the process from, such as Search. In that directory you should find an empty temp file named
Delete this temp file and you should be able to restart
fill_summary_index.py usage and commands
In the CLI, start by entering:
...and add the required and optional fields from the table below.
<boolean> options accept the values
yes for "true" and
no for "false."
||Earliest time (required). Either a UTC time or a relative time string.|
||Latest time (required). Either a UTC time or a relative time string.|
||The app context to use (defaults to |
||Specify a single saved search name. Can specify multiple times to provide multiple names. Use the wildcard symbol ("*") to specify all enabled, scheduled saved searches that have a summary index action.|
||Specify a comma seperated list of saved search names.|
||Specify a file with a list of saved search names, one per line. Lines beginning with a |
||The user context to use (defaults to "None").|
||Identifies the summary index that the saved search populates. If the index is not provided, the backfill script tries to determine it automatically. If this attempt at auto index detection fails, the index defaults to "summary".|
||The authentication string expects either |
||Number of seconds to sleep between each search. Default is 5 seconds.|
||Maximum number of concurrent searches to run (default is 1).|
||When this option is set to true, the script does not run saved searches for a scheduled timespan if data already exists in the summary index for that timespan. This option is set to false by default.|
Note: This option has no connection to the
||Specifies that the summary indexes are not on the search head but are on the indexes instead, if you are working with a distributed environment. To be used in conjunction with |
||When this option is set to true, the script periodically shows the done progress for each currently running search that it spawns. If this option is unused, its default is false.|
|Advanced options: these should not be used in almost all cases.|
||When this option is set to false, the script runs each search but does not trigger the summary indexing action. If this option is unused its default is true.|
||Indicates the search to be used to determine if data corresponding to a particular saved search at a specific scheduled times is present.|
||Same as |
||Indicates the field in the summary index data that contains the name of the saved search that generated that data.|
||Indicates the field in the summary index data that contains the scheduled time of the saved search that generated that data.|
Use summary indexing for increased reporting efficiency
Configure summary indexes
This documentation applies to the following versions of Splunk® Enterprise: 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11, 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12