About report acceleration and summary indexing
Splunk is capable of generating reports on massive amounts of data. However, the amount of time it takes to compute such reports is directly proportional to the number of events they summarize. Plainly put, it can take a lot of time to report on very large data sets. If you only have to do this on an occasional basis, the length of time may not be an issue. But running such reports on a regular schedule (or using them as the basis for panels in popular dashboards) is impractical--and this impracticality only increases exponentially as more and more users in your organization use Splunk to run similar reports.
To efficiently report on large volumes of data, you need to create data summaries that are populated by the results of background runs of the search. When you next run the search against data that has been summarized in this manner, it should complete significantly faster, because the summaries are much smaller than the original events from which they were generated.
Splunk provides two data summary creation methods: report acceleration and summary indexing.
Report acceleration is the simplest of the two methods of data-summary-backed search acceleration; setting it up for a large dataset search is as easy as clicking a checkbox and setting a time range. Future runs of the search should run faster as long as they're run (at least partially) within this time range. Report acceleration is preferable over summary indexing for the following reasons:
- Kicking off report acceleration is as easy as clicking a checkbox and selecting a time range. Everything after that happens behind the scenes. For summary indexing you need to design a search to populate the index that includes special search commands; you may need to create the summary index as well.
- Splunk automatically shares report acceleration summaries with similar searches. Say an employee named Mary sets up report acceleration for a search, which leads to Splunk building a summary for it. Then, a few days later, Joe designs a search that is nearly identical to Mary's search, with a few variations. When Joe turns on report acceleration for the search and saves it, Splunk automatically assigns it to the summary that was already built for Mary's search, which means that Joe won't need to wait for the summary to be built.
- Report acceleration features automatic backfill. If for some reason you have a data interruption, Splunk can detect this and automatically update or rebuild your summaries as appropriate.
- Report acceleration summaries are stored alongside the buckets in your indexes. Summary indexes, on the other hand, reside on the search head. Storing summaries in indexes at the bucket level enables Splunk to easily handle the dilemma of late-arriving events--something that can force full rebuilds of summary indexes. Because Splunk summaries can simultaneously span both hot and warm buckets, they can effortlessly summarize late-arriving data, because such data can only be added to hot buckets.
It's important to note that not all searches qualify for report acceleration: only searches that utilize reporting commands--searches that generate reports in the form of tables and charts--are eligible. In addition, any commands used in the search before the reporting command must be streaming commands. This limitation is related to the fact that the summaries are built at the index level rather than the search head.
You can enable report acceleration for an eligible search when you save it or add it to a dashboard in the Splunk Web UI. You can also enable report acceleration for an eligible search in Manager > Searches and Reports. For more information, see "Save searches and share search results" in this manual.
You use the Report acceleration summaries page in Manager to review and manage the summaries created through report acceleration. For more information about this, see "Manage report acceleration" in this manual. This topic also explains how summaries work and includes examples of qualifying and non-qualifying searches.
When should I use report acceleration?
Report acceleration is good for just about any slow-completing search that has 100k or more hot bucket events and which meets the qualifying conditions outlined above.
For more information and examples of qualifying and nonqualifying searches see "Manage report acceleration" in this manual.
Summary indexing is a method you can use to speed up long-running searches that don't qualify for report acceleration, such as searches that use commands that are not streamable before the reporting command. It's similar to report acceleration in that it involves populating a data summary with the results of a search, but in this case the data summary is actually a special summary index that is built and stored on the search head. This summary index is populated by a scheduled search that is based on the search that you'd like to accelerate and which has Enable selected for summary indexing in Manager > Searches and Reports.
For example, if the search you want to accelerate uses a reporting command, you can populate its summary index with a search that swaps the reporting command with a similar "si-" prefix summary indexing reporting command:
There are two topics on summary indexing setup, both in this manual.
- "Use summary indexing for increased reporting efficiency" shows you the easy way of setting up summary indexes, with scheduled searches that use
- "Configure summary indexes" covers the tricky and difficult method of summary index setup with
overlapcommands. You should only use this latter method if you're comfortable setting up searches that take aggregated statistics into account.
When should I use summary indexing?
If the search you're using qualifies for report acceleration, it's almost always preferable to use that method of speeding up the performance of large data volume searches.
You might want to use summary indexing instead of report acceleration if:
- The primary search you want to accelerate includes nonstreamable commands before a reporting command (just as with report acceleration, searches that populate summary indexes must involve reporting commands).
- You would like to run any search against a particular summary index, simply by including
index=<summary_index_name>in your search string. (Under report acceleration, Splunk automatically decides whether or not a search can run against a specific data summary.)
Define navigation to saved searches and reports
Manage report acceleration
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18