Experiments overview

Introduced in version 3.2 of the Splunk Machine Learning Toolkit (MLTK), Experiments manage the data source, algorithm used and any additional parameters to configure that algorithm, within one framework. An Experiment is an exclusive knowledge object in Splunk that keeps track of its settings and history, as well as its affiliated alerts and scheduled trainings. Experiment Assistants enable the creation of a machine learning model through a guided user interface.

Through the Assistants workflow, users specify data sources, select an algorithm and algorithm parameters, select the fields for the algorithms to analyze and set training/test data splits. Once the user instructs the Assistant to fit the algorithms to the selected training data and generate results, the workflow continues through the available visualizations and statistical analysis. Every step of the workflow has tooltips as additional guides, the option to see the SPL being written by the Assistant (with an explanation for the commands), and an option to open a clone of the SPL in a new search window for customization by the user.

Choose the Experiment Assistant to suit your needs

Experiments cover six useful areas. Click below to view a more detailed workflow of a particular Experiment area:

• The Predict Numeric Fields Experiment Assistant uses regression algorithms to predict or estimate numeric values. Such models are useful for determining to what extent certain peripheral factors contribute to a particular metric result. After the regression model is computed, you can use these peripheral values to make a prediction on the metric result.

• The Predict Categorical Fields Experiment Assistant displays a type of learning known as classification. A classification algorithm learns the tendency for data to belong to one category or another based on related data.

• The Detect Numeric Outliers Experiment Assistant determines values that appear to be extraordinarily higher or lower than the rest of the data. Identified outliers are indicative of interesting, unusual, and possibly dangerous events. This assistant is restricted to one numeric data field.

• The Detect Categorical Outliers Experiment Assistant identifies data that indicate interesting or unusual events. This assistant allows non-numeric and multi-dimensional data, such as string identifiers and IP addresses. To detect categorical outliers, input data and select the fields for which to look for unusual combinations or a coincidence of rare values. When multiple fields have rare values, the result is an outlier.

• The Forecast Time Series Experiment Assistant forecasts the next values in a sequence for a single time series. The result includes both the forecasted value and a measure of the uncertainty of that forecast. Forecasting refers to the use of past time series data trends to make a prediction about likely future values.

• The Cluster Numeric Events Experiment Assistant partitions events with multiple numeric fields into groups of events based on the values of those fields. The groupings aren't known in advance an the algorithms are often referred to as unsupervised learning.

Experiment composition

Each Experiment contains the following sections that will vary slightly depending on the type of machine learning analytic being performed:

• Create or Detect: Follow the workflow laid out in the Experiment to create a new model or forecast, or detect outliers. The workflow depends on the type of analytic but usually includes performing a lookup on a dataset, selecting a field to predict or analyze, and selecting fields or values to use for performing different types of analysis.
• Raw Data Preview: This section is displayed for predictions and forecasts to show you the data that is being used.
• Validate: Use the tables and visualizations to determine how well the model was fitted, how well outliers were detected, or how well a forecast performed.
• Deploy: Click the buttons beneath the visualizations and tables to see different ways to use the analysis. For example, you can open the search in the Search app, show the SPL, or create an alert.
• Experiment History tab: Each time you use an Experiment, a history is captured of the settings used. Compare the effects of different searches, algorithms and parameters, and identify the best choices for your use.

Experiment commands

Commands can vary depending upon which Assistant is selected:

• Predict Numeric and Predict Categorical Fields use the fit and apply commands
• Forecast Time Series can use the predict command
• Cluster Numeric Events uses the fit and apply commands