Manage summary index gaps and overlaps

Manage summary index gaps and overlaps

The accuracy of your summary index searches can be compromised if the summary indexes involved have gaps or overlaps in their collected data.

Gaps in summary index data can come about for a number of reasons:

• A summary index initially only contains events from the point that you start data collection: Don't lose sight of the fact that summary indexes won't have data from before the summary index collection start date--unless you arrange to put it in there yourself with the backfill script.
• splunkd outages: If splunkd goes down for a significant amount of time, there's a good chance you'll get gaps in your summary index data, depending on when the searches that populate the index are scheduled to run.
• Searches that run longer than their scheduled intervals: If the search you're using to populate the scheduled search runs longer than the interval that you've scheduled it to run on, then you're likely to end up with gaps because Splunk won't run a scheduled search again when a preceding search is still running. For example, if you were to schedule the index-populating search to run every five minutes, you'll have a gap in the index data collection if the search ever takes more than five minutes to run.

Overlaps are events in a summary index (from the same index-populating search) that share the same timestamp. Overlapping events skew reports and statistics created from summary indexes. Overlaps can occur if you set the time range of a saved search to be longer than the scheduled search interval. In other words, don't arrange for an hourly search to gather data for the past 90 minutes.

Note: For general information about creating and maintaining summary indexes, see "Use summary indexing for increased reporting efficiency" in the Knowledge Manager manual.

Use the backfill script to add other data or fill summary index gaps

The fill_summary_index.py script backfills gaps in summary index collection by running the saved searches that populate the summary index as they would have been executed at their regularly scheduled times for a given time range. In other words, even though your new summary index only started collecting data at the start of this week, if necessary you can use fill_summary_index.py to fill the summary index with data from the past month.

In addition, when you run fill_summary_index.py you can specify an App and schedule backfill actions for a list of summary index searches associated with that App, or simply choose to backfill all saved searches associated with the App.

When you enter the fill_summary_index.py commands through the CLI, you must provide the backfill time range by indicating an "earliest time" and "latest time" for the backfill operation. You can indicate the precise times either by using relative time identifiers (such as -3d@d for "3 days ago at midnight") or by using UTC epoch numbers. The script automatically computes the times during this range when the summary index search would have been run.

NOTE: To ensure that the fill_summary_index.py script only executes summary index searches at times that correspond to missing data, you must use -dedup true when you invoke it.

The fill_summary_index.py script requires that you provide necessary authentication (username and password). If you know the valid Splunk key when you invoke the script, you can pass it in via the -sk option.

The script is designed to prompt you for any required information that you fail to provide in the command line, including the names of the summary index searches, the authentication information, and the time range.

Examples of fill_summary_index.py invocation

If this is your situation:

You need to backfill all of the summary index searches for the splunkdotcom App for the past month--but you also need to skip any searches that already have data in the summary index:

Then you'd enter this into the CLI:

./splunk cmd python fill_summary_index.py -app splunkdotcom -name "*" -et -mon@mon -lt @mon -dedup true -auth admin:changeme

If this is your situation:

You need to backfill the my_daily_search summary index search for the past year, running no more than 8 concurrent searches at any given time (to reduce impact on Splunk performance while the system collects the backfill data). You do not want the script to skip searches that already have data in the summary index. The my_daily_search summary index search is owned by the "admin" role.

Then you'd enter this into the CLI:

./splunk cmd python fill_summary_index.py -app search -name my_daily_search -et -y -lt now -j 8 -owner admin -auth admin:changeme

Note: You need to specify the -owner option for searches that are owned by a specific user or role.

What to do if fill_summary_index.py is interrupted while running

In the app that you are invoking fill_summary_index.py from (default: 'search'), there will be a 'log' directory. In this directory, there will be an empty temp file named 'fsidx*lock'.

Delete the 'fsidx*lock' file and you will be able to restart fill_summary_index.py.

fill_summary_index.py usage and commands

In the CLI, start by entering:

python fill_summary_index.py

...and add the required and optional fields from the table below.

Note: <boolean> options accept the values 1, t, true, or yes for "true" and 0, f, false, or no for "false."

Field	Value
-et <string>	Earliest time (required). Either a UTC time or a relative time string.
-lt <string>	Latest time (required). Either a UTC time or a relative time string.
-app <string>	The application context to use (defaults to None).
-name <string>	Specify a single saved search name. Can specify multiple times to provide multiple names. Use the wildcard symbol ("*") to specify all enabled, scheduled saved searches that have a summary index action.
-names <string>	Specify a comma seperated list of saved search names.
-namefile <filename>	Specify a file with a list of saved search names, one per line. Lines beginning with a # are considered comments and ignored.
-owner <string>	The user context to use (defaults to "None").
-index <string>	Identifies the summary index that the saved search populates. If the index is not provided, the backfill script tries to determine it automatically. If this attempt at auto index detection fails, the index defaults to "summary".
-auth <string>	The authentication string expects either <username> or <username>:<password>. If only a username is provided, the script requests the password interactively.
-sleep <float>	Number of seconds to sleep between each search. Default is 5 seconds.
-j <int>	Maximum number of concurrent searches to run (default is 1).
-dedup <boolean>	When this option is set to true, the script doesn't run saved searches for a scheduled time if data already exists in the summary index. If this option is not used, its default is false.
-showprogress <boolean>	When this option is set to true, the script periodically shows the done progress for each currently running search that it spawns. If this option is unused, its default is false
Advanced options: these should not be used in almost all cases
-trigger <boolean>	When this option is set to false, the script runs each search but does not trigger the summary indexing action. If this option is unused its default is true.
-dedupsearch <string>	Indicates the search to be used to determine if data corresponding to a particular saved search at a specific scheduled times is present
-namefield <string>	Indicates the field in the summary index data that contains the name of the saved search that generated that data.
-timefield <string>	Indicates the field in the summary index data that contains the scheduled time of the saved search that generated that data

Use the overlap command to identify summary index gaps and overlaps

To identify gaps and overlaps in your data, run a search against the summary index that uses the overlap command. This command identifies ranges of time in the index that include gaps or overlaps. If you suspect that a particular time range might include gaps and/or overlaps, you can identify it in the search by specifying a start time and end time or a period and a saved search name, following the | overlap command in the search string.

Use these two commands to define a specific calendar time range:

• StartTime: Time to start searching for missing entries, starttime= mm/dd/yyyy:hh:mm:ss (for example: 05/20/2008:00:00:00).
• EndTime: Time to stop searching for missing entries, endtime= mm/dd/yyyy:hh:mm:ss (for example: 05/22/2008:00:00:00).

Or use these two commands to define a period of time and the saved search to search for missing events with:

• Period: Specify the length of time period to search, period=<integer>[smhd] (for example: 5m).
• SavedSearchName: Specify the name of the saved search to search for missing events with savedsearchname=string (NO wildcards).

If you identify a gap, you can run your scheduled saved search over the period of the gap and summary index the results with the backfill script (see below).

If you identify overlapping events, you can manually delete the overlaps from the summary index by using the search language.

• Was this topic useful?
• Post a comment