Manage report acceleration
Report acceleration is the easiest way to speed up report-creating searches that take a long time to complete because they have to cover a large volume of data. You enable acceleration for a reporting search when you save it or create a dashboard panel based on it via Splunk Web.
This topic covers various aspects of report acceleration in more detail. It includes:
- A quick guide to enabling automatic acceleration for a reporting search.
- Examples of qualifying and nonqualifying searches (only specific kinds of searches qualify for report acceleration).
- Details on how Splunk creates and maintains report acceleration summaries.
- An overview of the Report acceleration summaries page in Manager, which you can use to review and maintain the data summaries that Splunk uses for automatic report acceleration.
Enabling report acceleration
To enable report acceleration for a slow-running search, all you have to do is:
1. In the Search App, run the search.
2. Click Save and select Save search... OR click Create and select Dashboard panel...
3. If you see Turn on acceleration, click it. (In the Create Dashboard Panel dialog box, the Turn on acceleration switch will appear in the Panel step.)
Note: You won't see Turn on acceleration if:
- your permissions don't enable you to accelerate reports. Your role should have the capability to schedule searches.
- your search does not qualify for report acceleration. For more information, see the subtopic "How searches qualify for acceleration," below.
4. The Summary range field should appear. Select from 1 Day, 7 Days, 1 Month, 3 Months, 1 Year, or All Time depending on the range of time over which you plan to run the search. For example, if you only plan to run the search over periods of time within the last seven days, choose 7 Days.
5. Save the search or panel with whatever other settings you require.
Once you save the search Splunk will begin building the summary if it determines that the search would benefit from summarization. See the subtopic "Conditions under which Splunk will not build a summary," below, if you save the search and find that its summary is not being constructed (if you go to Manager > Report Acceleration Summaries, you'll notice that the Summary Status is stuck at 0% complete for an extended amount of time).
Once Splunk builds the summary, future runs of this search should complete faster than they did before. See the subtopics below for more information on summaries and how they work.
Note: When you are running searches in the Timeline view, keep in mind that report acceleration only works for searches that have Search Mode set to Smart or Fast. If you select the Verbose search mode for a search that benefits from report acceleration, it will run as slow as it would if no summary existed for it. ('Search Mode does not affect searches powering dashboard panels.) For more information about the Search Mode settings, see "Set search mode to adjust your search experience" in the Search Manual.
Alternatively you can enable report acceleration for an existing search at Manager > Searches and reports.
For a more thorough description of this procedure, see "Save searches and share search results," in this manual.
How searches qualify for acceleration
For a search to qualify for acceleration it must use a reporting command (such as
top)--this is why we call this feature "report acceleration."
In addition, if there are any other commands before the first reporting search they must be streamable, which means that they apply a transformation to each event returned by the search. For example, the
rex command is streamable; it is applied to each event returned by the search and extracts fields when they match the regular expression provided with the command. Other common streaming commands used for reporting include
bin (as long as you provide an explicit span), and
lookup (as long as it is not
Note: You can use nonstreaming commands after the first reporting command and still have the search qualify for automatic acceleration. It's just nonstreaming commands before the first reporting command that disqualify the search.
Examples of qualifying searches
Here are examples of search strings that qualify for report acceleration:
index=_internal | stats count by sourcetype
index=_audit search=* | rex field=search "'(?.*)'" | chart count by user search
test foo bar | bin _time span=1d | stats count by _time x y
index=_audit | lookup usertogroup user OUTPUT group | top searches by group
Examples of nonqualifying searches
And here are examples of search strings that do not qualify for report acceleration:
Reason the following search fails: This is a simple event search, with no reporting command.
index=_internal metrics group=per_source_thruput
Reason the following search fails:
eventstats is not a reporting command.
index=_internal | eventstats avg(x) by y
Reason the following search fails:
transaction is not a streaming command. Other non-streaming commands include
tail, and any other search command that is not on the list of streaming commands.
index=_internal | transaction user maxspan=30m | timechart avg(duration) by user
Searches that qualify for report acceleration but won't get much out of it
In addition, you can have searches that technically qualify for report acceleration, but which may not be helped much by it. This is often the case with searches with high data cardinality--something you'll find when there are two or more reporting commands in the search and the first reporting command generates many (50k+) output rows. For example:
index=* | stats count by id | stats avg(count) as avg, count as distinct_ids
Understand report acceleration summaries
The following subtopics cover various aspects of report summaries to give you a better idea of how they work.
Set report acceleration summary time ranges
When you enable report acceleration for a search, Splunk can begin generating a summary that spans a specific range of time, such as the past day, the past seven days, the past month, the past year, and so on. You determine this time range when you choose a value from the Summary range list.
For example, if you set a summary range of 1 week, Splunk can create a data summary for the search that covers the past seven days at any point in time, approximately. Once the summary fully built, going forward the report summarization process ensures that the summary always covers the selected range, removing older summary data that passes out of the range.
Note: We say that the Summary Range indicates the approximate range of time that a summary spans. At times, summaries will have a store of data that that slightly exceeds their summary range, but they never fail to meet it, unless it's the first time the summary is being built.
In the future, when you run the accelerated search over a range that falls within the preceding week, Splunk will run the search against its summary rather than the source index (the index the search originally searched against). In most cases the summary will have far less data than the source index, and that--along with the fact that the report summary already contains precomputed results for portions of the search pipeline--means that the search should complete faster than it did on its initial run.
If you run the accelerated search over a period of time that is only partially covered by the summary, the search won't complete quite as fast. This is because Splunk has to run at least part of the search over raw data in the main Splunk index. For example, if the Summary Range setting for a search is 1 week and you run the search over the last 9 days, Splunk only gets acceleration benefits for the portion of the search that covers the past 7 days. The portion of the search that runs over days 8 and 9 will run at normal speed.
Keep this in mind when you set the Summary Range value. If you always plan to run a search over time ranges that exceed the past 7 days, but don't extend further out than 30 days, you should select a Summary Range of 1 month when you set up report acceleration for that search.
Note: Splunk usually only generates a summary for a search when the number of events in the hot bucket covered by the chosen Summary Range is equal to or greater than 100k. For more information, see the subtopic below titled "Conditions under which Splunk will not build or update a summary."
How Splunk builds summaries
When you enable acceleration for an eligible search and Splunk determines that it will build a summary for the search, Splunk begins running the search to populate the summary with data. When the summary is complete, Splunk continues running the search on a 10 minute interval to keep the summary up to date. Each update ensures that the entire configured time range is covered by a summary. This method of summary building also ensures that late-arriving data will be summarized without complication.
It can take Splunk some time to build a report summary. The creation time depends on the number of events involved, the overall summary range, and the length of the summary timespans (chunks) in the summary.
You can track progress toward summary completion on the Report Acceleration Summaries page in Manager. On the main page you can check the Summary Status to see what percentage of the summary is complete.
Note: Just like ordinary scheduled searches, the searches that automatically populate report acceleration summaries on a regular schedule are managed by the search scheduler. The search scheduler is allowed to allocate up to 25% of its total search bandwidth for report acceleration summary creation.
The search scheduler also runs searches that populate report acceleration summaries at the lowest priority. If these "auto-summarization" searches have a scheduling conflict with user-defined alerts, summary-index searches, and regular scheduled searches, the user-defined searches always get run first. This means that you may run into situations where a summary isn't being created or updated because Splunk is busy running more prioritized searches.
For more information about the search scheduler see the topic "Configure the priority of scheduled searches," in this manual.
Summary data is divided into chunks with regular timespans
As Splunk builds and maintains the summary, it breaks the data up into chunks to ensure statistical accuracy, according to a "timespan" that Splunk determines automatically based on the overall summary range. For example, if the summary range for a search is 1 month then Splunk will likely give it a timespan of 1d (one day).
A summary timespan represents the smallest time range for which the summary contains statistically accurate data. If you are running a search against a summary that has a one hour timespan, the time range you choose for the search should be evenly divisible by that timespan if you want the search to use the summarized data. When you are dealing with a 1h timespan, a search over the past 24 hours would work fine, but a search over the past 90 minutes might not be able to use the summarized data.
Summaries can have multiple timespans
Splunk will give report acceleration summaries multiple timespans if necessary to make them as searchable as possible. For example, a summary with a summary range of 3 months can have timespans of 1mon and 1d. In addition, Splunk may assign extra timespans when the summary spans more than one index bucket and the buckets cover very different amounts of time. For example, if a summary spans two buckets, and the first bucket spans two months and the next bucket spans 40 minutes, the summary will have chunks with 1d and 1m timespans.
You can manually set summary timespans (but we don't recommend it)
As we mentioned before, Splunk determines timespans for summaries automatically, but you can adjust them manually at the search level in
savedsearches.conf (look for the
auto_summarize.timespan parameter). If you do set your summary timespans manually, keep in mind that very small timespans can result in extremely slow summary creation times, especially if the summary range is long. On the other hand, large timespans can result in quick-building summaries that cannot not be used by searches with short time ranges. In almost all cases, for optimal performance and usability it's best to let Splunk determine summary timespans.
Where report acceleration summaries are created and stored
Splunk creates each accelerated report summary on the indexer, parallel to the bucket or buckets that cover the range of time over which the summary spans.
It is important to keep in mind that report summaries are not indexes. Report summaries are not covered by clustering; if you are using clustering and the primary bucket is no longer available the summaries will not be available until the next time Splunk runs summary update searches, finds that the summaries are missing, and regenerates them.
Note: Until Splunk regenerates your summaries, the results of your accelerated reports will be correct, but they will run slower.
Multiple searches for a single summary
A single report summary can be associated with multiple searches when the searches meet the following two conditions.
- The searches are identical up to and including the first reporting command.
- The searches have the same owner and are associated with the same app.
Searches that meet the first condition, but which have different owners or belong to different apps, cannot share the same summary.
For example, these two saved searches use the same report acceleration summary.
sourcetype=access_* status=2* | stats count by price
sourcetype=access_* status=2* | stats count by price | eval discount = price/2
These two searches use different report acceleration summaries.
sourcetype=access_* status=2* | stats count by price
sourcetype=access_* status=2* | timechart by price
You can also run non-saved searches against the summary, as long as the basic search matches the populating saved search up to the first reporting command and the search time range fits within the summary span. You can see which searches are associated with your summaries by navigating to Manager > Report Acceleration Summaries. See "Use the Report Acceleration Summaries Page" in this topic.
Conditions under which Splunk Enterprise cannot build or update a summary
Splunk Enterprise cannot build a summary for a search when the data you want it to summarize meets the following conditions:
- The number of events in the hot bucket covered by the chosen Summary Range is less than than 100k. You see a Summary Status warning that says Not enough data to summarize when this condition exists.
- Splunk Enterprise estimates that the completed summary will not exceed 10% of the total size of the total bucket size in your deployment. When Splunk Enterprise makes this estimation, it suspends the summary for 24 hours (you will see a Summary Status of Suspended).
You can see the Summary Status for a summary in Settings > Report Acceleration Summaries.
If you define a summary and Splunk Enterprise does not create it because these conditions are met, it continues to periodically check to see if conditions improve. When Splunk Enterprise finds that the summary has 100k or more hot bucket events in its range and will not be overlarge when complete, it begins creating or updating the summary.
How can I tell if a search is using its summary?
The obvious clue that a search is using its summary is if you run it and find that its search performance has improved (it completes faster than it did before).
But if that's not enough, or if you aren't sure if there's a performance improvement, you can check the Search Job Inspector for a debug message that indicates that the search is using a specific report acceleration summary. Here's an example:
DEBUG: [thething] Using summaries for search, summary_id=246B0E5B-A8A2-484E-840C-78CB43595A84_search_admin_b7a7b033b6a72b45, maxtimespan=
In this example, that last string of numbers,
b7a7b033b6a72b45, corresponds to the Summary ID displayed on the Report Acceleration Summaries page.
Use the Report Acceleration Summaries page
You can review the your report acceleration summaries and even manage various aspects of them with the Report Acceleration Summaries page in Manager. Go to Manager > Report Acceleration Summaries.
The main Report Acceleration Summaries page enables you to see basic information about the summaries that you have permission to view.
The Summary ID is a unique string of numbers and letters that Splunk assigns to each summary. It is derived from the search string and is used as part of the name of the directory that Splunk creates for the summary files. Click a summary ID to view summary details and perform summary management actions. For more information about this detail view, see the subtopic "Review summary details," below.
The Reports Using Summary column lists the saved reports that are associated with each of your summaries. It indicates that each search associated with a particular summary will get search acceleration benefits from that summary. Click on a report title to drill down to the detail page for that report.
Check Summarization Load to get an idea of the effort that Splunk has to put into updating the summary. It's calculated by dividing the number of seconds it takes to run the populating search by the interval of the populating search. So if the search runs every 10 minutes (600 seconds) and takes 30 seconds to run, the summarization load is 0.05. If the summarization load is high and the Access Count for the summary shows that the summary is rarely used or hasn't been used in a long time, you might consider deleting the summary to reduce the strain on your system.
The Summary Status column reports on the general state of the summary and tells you when it was last updated with new data. Possible status values are Complete, Pending, Suspended, Not enough data to summarize, or the percentage of the summary that is complete at the moment. If you want to update a summary to the present moment, click its summary ID to go to its detail page and click Update to kick off a new summary-populating search.
If the Summary Status is Pending it means that the associated summary was last updated more than 10 minutes ago and is due to be updated again. It's saying that the summary may be slightly outdated but will be updated momentarily.
If the Summary Status is Suspended it means that Splunk has determined that the search is not worth summarizing because it will create a summary that is too large. Splunk projects the size of the summary that a search will create--if it determines that the summary will be larger than 10% of the index buckets it spans, it will suspend the summary for 24 hours. There's no point to creating a summary, for example, if the summary contains 90% of the data in the full index.
You cannot override summary suspension, but you can adjust the length of time that summaries are suspended by changing the value of the
auto_summarize.suspend_period attribute in
If the Summary Status reads Not enough data to summarize, it means that Splunk is not currently generating or updating a summary because the searches associated with it are returning less than 100k events from the hot buckets covered by the summary range. For more information, see the subtopic above titled "Conditions under which Splunk will not build or update a summary."
Review summary details
You use the summary details page to view detail information about a specific summary and to initiate actions for that summary. You get to this page by clicking a Summary ID on the Report Acceleration Summaries page in Manager.
Under Summary Status you'll see basic status information for the summary. It mirrors the Summary Status listed on the Report Acceleration Summaries page (see above) but also provides information about the verification status of the summary.
If you want to update a summary to the present moment click the Update button under Actions to kick off a new summary-populating search.
No verification status will appear here if you've never initiated verification for the summary. After you initiate verification this status shows the verification percentage complete. Otherwise this status shows the results of the last attempt at summary verification; the possible values are Verified and Failed to verify, with an indication of how far back in the past this attempt took place.
For more information about summary verification, see "Verify a summary," below.
Reports using the summary
The Reports Using This Summary section lists the reports that are associated with the summary, along with their owner and home app. Click on a search title to drill down to the detail page for that search. Similar searches (searches that all transform the same root search with different reporting commands, for example) can use the same summary.
The Details section provides a set of metrics about the summary.
Summarization Load and Access Count are mirrored from the main Report Acceleration Summaries page. See the subtopic "Use the Report Acceleration Summaries page," above, for more information.
Size on Disk shows you how much space the summary takes up in terms of storage. You can use this metric along with the Summarization Load and Access Count to determine which summaries ought to be deleted.
Note: If the Size value stays at 0.00MB it means that Splunk is not currently generating this summary because the searches associated with it either don't have enough events (at least 100k hot bucket events are required) or the projected summary size is over 10% of the bucket with which the search is associated. Splunk will periodically check this search and automatically create a summary for it when it meets the criteria for summary creation.
Summary range is the range of time spanned by the summary, always relative to the present moment. You set this up when you define the search that populates the summary. For more information, see the subtopic "Set report acceleration summary time ranges," above.
Timespans displays the size of the data chunks that make up the summary. A summary timespan represents the smallest time range for which the summary contains statistically accurate data. So if you are running a search against a summary that has a one hour timespan, the time range you choose for the search should be evenly divisible by that timespan if you want to get good results. So if you are dealing with a 1h timespan, a search over the past 24 hours would work fine, but a search over the past 90 minutes might be problematic. See the subsection "How Splunk builds summaries," above, for more information.
Buckets' shows you how many index buckets the summary spans, and Chunks tells you how many data chunks comprise the summary. Both of these metrics are informational for the most part, though they may aid with troubleshooting issues you may be encountering with your summary.
Verify a summary
At some point you may find that an accelerated reporting search seems to be returning results that don't fit with the results the search returned when it was first created. This can happen when certain aspects of the search change without your knowledge, such as a change in the definition of a tag, event type, lookup, or field extraction rule used by the search. (When these kinds of things are changed, Splunk cannot detect them, and it will not automatically rebuild the summary to correct the summarized data.)
If you suspect that this has happened with one of your accelerated reports, go to the detail page for the summary with which your search is associated, you can run a verification process that examines a subset of the summary and verifies that all of the examined data is consistent. If it finds that the data is inconsistent, it notifies you that the verification has failed.
For example, say you have a search that uses an event type,
netsecurity, which is associated with a specific kind of network security event. You enable acceleration for this search, and Splunk builds a summary for it. At some later point, the definition of the event type
netsecurity is changed, so it finds an entirely different set of events, which means your summary is now being populated by a different set of data than it was before. You notice that the results being returned by the accelerated search seem to be different, so you run the verification process on it from the Report acceleration summaries page in Manager. The summary fails verification, so you begin investigating the root search to find out what happened.
Ideally the verification process should only have to look at a subset of the summary data in order to save time; a full verification of the entire summary will take as long to complete as the building of the summary itself. But in some cases a more thorough verification is required.
Clicking Verify opens a Verify Summary dialog box. Verify Summary provides two verification options:
- A Fast verification, which is set to quickly verify a small subset of the summary data at the cost of thoroughness.
- A Thorough verification, which is set to thoroughly review the summary data at the cost of speed.
In both cases, the estimated verification time is provided.
After you click Start to kick off the verification process you can follow its progress on the detail page for your summary under Summary Status. When the verification process completes, this is where you'll be notified whether it succeeded or failed. Either way you can click the verification status to see details about what happened.
When verification fails, the Verification Failed dialog can tell you what went wrong:
During the verification process, Splunk will skip hot buckets and buckets that are in the process of building.
When a summary fails verification you can review the root search (or searches) to see if it can be fixed to provide correct results. Once the search is working, click Rebuild to rebuild the summary so it is entirely consistent. Or, if you're fine with the search as-is, just rebuild the search. And if you'd rather start over from scratch, delete the summary and start over with an entirely new search.
Update, rebuild, and delete summaries
Click Update if the Summary Status shows that the summary has not been updated in some time and you would like to make it current. Update kicks off a standard summary update search to pull in events so that it is not missing data from the last few hours (for example).
Note: When a summary's Summary Status is Suspended, you cannot use Update to bring it current.
Click Rebuild to make Splunk rebuild the index from scratch. You may want to do this in situations where you suspect there has been data loss due to a system crash or similar mishap, or if it failed verification and you've either fixed the underlying search(es) or have decided that the summary is ok with the data it is currently bringing in.
Click Delete to remove the summary from the system (and not regenerate summaries in the future). You may want to do this if the summary is used infrequently and is taking up space that could better be used for something else. You can use the Searches and Reports page in Manager to reenable report acceleration for the search or searches associated with the summary.
About report acceleration and summary indexing
Use summary indexing for increased reporting efficiency
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18