By default, a Splunk search retrieves all events. However in some situations you might want to retrieve a sample set of events, instead of retrieving the entire event set. There are several reasons why you might want to use event sampling.
- To perform a quick search to ensure the correct events are being returned
- To determine the characteristics of a large data set without processing every event
- To test that the data selection, formatting, calculations, and other components of the search are working correctly
For most searches, event sampling can greatly increase search performance without decreasing functionality.
The event sampling ratio
The sampling ratio is the likelihood of any event being included in the sample result set. The formula for the ratio is
For example, if the sample ratio value is 100, each event has a 1 in 100 chance of being included in the result set. The selection of each event is independent of the selection of all another events. It is possible that many events are included from the first 100 events, or none at all.
If a search matches 1,000,000 events when sampling is not used, using a sample ratio value of 100 would result in returning approximately 10,000 events.
If you to rerun a sampling search many times, the exact number of returned results is modeled by a binomial distribution with n=1000000 and p=0.01. This distribution looks like a normal distribution, with the mean=10000 and the standard deviation (stdev)=99.5.
In Splunk Web, the sampling ratio that you specify must be a positive integer that is greater than 1. To disable sampling in Splunk Web, set the ratio to 1.
Set the default sampling ratio
- Splunk Cloud Platform
- To set the default sampling ratio, request help from Splunk Support. If you have a support contract, file a new case using the Splunk Support Portal at Support and Services. Otherwise, contact Splunk Customer Support.
- Splunk Enterprise
- Only users with file system access, such as system administrators, can set the default sampling ratio.
- Review the steps in How to edit a configuration file in the Admin Manual.
Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make the changes in the local directory.
- Open the local
ui-prefs.conffile for the Search app. For example,
- Under the [Preference options] stanza, set
display.prefs.customSampleRatioto a positive integer.
How event sampling works
By default, event sampling is not active. When you run a search, every event that matches your criteria is returned. When you specify a ratio, sampling remains in effect for the active search window. Sampling also remains in effect when you save a search as a report or dashboard panel.
When you specify a ratio value, your value overrides the default value configured for your Splunk deployment and remains in effect until you change it.
If you open a new search window, event sampling is no longer active. However, the last custom ratio that you used appears in the Sampling drop-down.
Commands and functions to avoid with event sampling
Typically, searches that use the
streamstats commands are not good candidates for sampling.
When you calculate statistics using a sample set of events, the statistical values will not be accurate. To get an approximation of the true statistical values, you must scale the values that the search returns. To get the most accurate value approximation, the scaling factor should be the same as the sampling ratio.
For example, you create a report using this search with event sampling enabled.
... | stats sum(x)
Because you used event sampling, the returned value is not the complete sum of all of the events. It is only the sum of the sample set of events. If the sampling ratio is 100, the true sum is approximately 100 times the value returned by the search.
Statistical calculations that fall into this situation are
Other statistics that are difficult to interpret when event sampling is used include:
Specify a sampling ratio
You activate event sampling for a search by specifying a sampling ratio.
1. In Splunk Web, below the Search bar, click No Event Sampling.
2. You can use one of the default ratios or specify a custom ratio.
- a. To use one of the default ratios, click the ratio in the Sampling drop-down.
- b. To specify a custom ratio, click Custom and type the ratio value. Then click Apply. The ratio value must be a positive integer greater than 1.
Event sampling indicators
There are several indicators in the Search & Reporting App window which show that event sampling is active. After you run a search, the Sampling drop-down appears in the event count line. The label for the Sampling drop-down specifies the ratio that is applied to the search. Additionally, if a sampling ratio is being used, the Jobs drop-down specifies the ratio that is applied to the search.
Event sampling with reports and dashboard panels
You can save a search that uses event sampling as a report or dashboard panel. Use the Save As drop-down to save the search.
When the search is saved as a report, the sampling ratio is used when the report is run.
When the search saved as a dashboard panel, the panel is powered by an inline search. When the dashboard is refreshed, the sampling ratio that was saved with the inline search is used.
If you open a report and add the report to a dashboard panel, you can specify how the panel is powered. You can specify that the panel is powered by the inline search that the report is based on. Or you can specify that the panel is powered by the report itself.
- Panels powered by reports
- When you view the source for the panel in Simple XML, there is no indication if the report uses event sampling.
- Panels powered by inline searches
- When you view the source for the panel in Simple XML, if the underlying search uses event sampling there is <sampleRatio> entry. For example:
<event> <title>sample events</title> <search> <query>buttercupgames</query> <earliest>@d</earliest> <latest>now</latest> <sampleRatio>500</sampleRatio> </search> </event>
- Accelerated reports
- You cannot accelerate reports that are based on event sampling searches. See Accelerate reports in the Reporting Manual.
Use fields to retrieve events
Retrieve events from indexes
This documentation applies to the following versions of Splunk® Enterprise: 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.2.0