Splunk® Machine Learning Toolkit

User Guide

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of Splunk® Machine Learning Toolkit. For documentation on the most recent version, go to the latest release.
Acrobat logo Download topic as PDF

Forecast Time Series Experiment workflow

Experiments manage the data source, algorithm, and the parameters to configure that algorithm within one framework. An Experiment is an exclusive knowledge object in the Splunk platform that keeps track of its settings and history, as well as its affiliated alerts and scheduled trainings. Experiment Assistants enable machine learning through a guided user interface.

The Forecast Time Series Assistant predicts the next value in a sequence of time series data. The result includes both the predicted value and a measure of the uncertainty of that prediction. Forecasting is one type of prediction. Forecasting refers to the use of past time series data to make predictions.

The following visualization shows a time series and the split between the training and testing data.

The visualization shows a time series and the split between the training and testing data

Algorithms

The Forecast Time Series Experiment Assistant uses the following algorithms:

  • State-space method using the Kalman filter
  • Autoregressive Integrated Moving Average (ARIMA)

Forecast Time Series

Input the data and select the field you want to forecast .

Workflow

Follow these steps to create a Forecast Time Series Experiment.

  1. From the MLTK navigation bar, click Experiments.
    • If this is the first experiment in the toolkit, you will land on a display screen of all six Assistants. Select the Forecast Time Series block.
    • If you have at least one Experiment in the toolkit, you will land on a list view of Experiments. Click the Create New Experiment button.
  2. Fill in an Experiment Title, and add a description. Both the name and description can be edited later if needed.
  3. Click Create.
  4. Run a search, and be sure to include a date range.
  5. Select an algorithm from the Algorithm drop-down menu.
    If you are not sure which algorithm to choose, start with the default Kalman filter algorithm.
  6. From the Field to Forecast list, select the field you want to forecast.
    The Field to Forecast drop-down menu populates with fields from the search.
  7. Select the parameters.
    If Then
    You chose the Kalman filter algorithm: These methods consider subsets of features such as local level (an average of recent values), trend (a slope of line that fits through recent values), and seasonality (repeating patterns).
    You chose the ARIMA algorithm: Specify the values for AR (autoregressive) - p, I (integrated) - d, MA (moving average) - q parameters.

    For example, AR(1) means you would forecast future values by looking at 1 past value. I(1) means it took 1 difference, where each data point was subtracted from the one that follows it, to make the time series stationary. MA(1) means you would forecast future values using 1 previous prediction error.

  8. Specify the Future Timespan, which indicates how far beyond the data you want to forecast.
    The size of the confidence interval is used to gauge how confident the algorithm is in its forecast.
  9. In the Holdback field, specify the number of values to withhold. Decide how many search results to use for validating the quality of the forecast.
    The larger the withholding, the fewer values available to train your model.
  10. Select the Confidence Interval, which is the percentage of the future data you expect to fall inside of the confidence envelope.
  11. For the Kalman algorithm, select the Period, which indicates the period of any known repeating patterns in the data to assist the algorithm.
    For example, if your data includes monthly sales data that follows annual patterns, specify 12 for the period.
  12. (Optional) Add notes to this Experiment. This free form block of text can help you track the selections made in the fields above. Refer back to notes to review which parameter combinations yield the best results. <note>Hover over any field name to get more information about that field</notes>
  13. Click Forecast. The Experiment is now in a Draft state. Draft versions allow you to alter settings without committing or overwriting a saved Experiment. An Experiment is not stored to Splunk until it is saved.
    The following table explains the differences between a draft and a saved Experiment.
    Action Draft Experiment Saved Experiment
    Create new record in Experiment history Yes No
    Run Experiment search jobs Yes No
    (As applicable) Save and update Experiment model No Yes
    (As applicable) Update all Experiment alerts No Yes
    (As applicable) Update Experiment scheduled trainings No Yes

Interpret and validate

After you forecast a time series, review your results in the following tables and visualizations.

Result Definition
Raw Data Preview This displays the raw data from the search.
Forecast This graphs displays the actual value as a solid line and the predicted value as a dotted line, surrounded by a confidence envelope.

Values that fall outside the confidence envelope are outliers. A vertical line indicates where training data stops and test data begins. When the real data ends, forecasted values are displayed in shades of green. The larger the envelope, the less confidence we have about forecasts around that time. The size of the envelope is directly related to the specified confidence interval percentage.

R2 Statistic This statistic explains how well the model explains the variability of the result. 100% (a value of 1) means the model fits perfectly. The closer the value is to 1 (100%), the better the result.
Root Mean Squared Error This statistic explains the variability of the result, essentially the standard deviation of the residual. The formula takes the difference between actual and predicted values, squares this value, takes an average, and then takes a square root. The result is an absolute measure of fit, the smaller the number the better the fit. These values only apply to one dataset and are not comparable to values outside of it.
Prediction Outliers This result shows the total number of outliers detected.

Predicting with the ARIMA algorithm

When predicting using the ARIMA algorithm, additional autocorrelation panels are present. Autocorrelation charts can be used to estimate and identify the three main parameters for the model:

  • the autoregressive component p
  • the integrated component or order of differencing d
  • the moving average component q

ACF: Autocorrelation function chart

The autocorrelation function chart shows the predicted field's autocorrelations at various lags, surrounded by confidence interval lines. For example, the column at lag 1 shows the amount of correlation between the time series and a lagged version of itself. Autocorrelation Function chart

PACF: Partial autocorrelation function chart

The partial autocorrelation function chart shows the predicted field's autocorrelations at various lags while controlling for the amount of correlation contributed by earlier lag points. This chart is also surrounded by confidence interval lines. For example, the column at lag 2 shows the amount of correlation between the time series and a lagged version of itself, while removing the correlation contributed by the lag 1 data points. Partial Autocorrelation Function chart

ACF Residual: Autocorrelation function residual chart

The autocorrelation function residual chart shows prediction errors. The errors are the difference between the series and the predictions. The ACF of the residuals should be close to zero. If the errors are highly correlated, the model might be poorly parameterized, or the series might not be stationary. Autocorrelation Function Residual chart

PACF Residual: partial autocorrelation function residual chart

The partial autocorrelation function chart shows prediction errors. The errors are the difference between the series and the predictions. The PACF of the residuals should be close to zero. If the errors are highly correlated, the model might be poorly parameterized or the series might not be stationary. Autocorrelation Function Residual chart

Refine the forecast

After you create a forecast, you can select a different algorithm to see whether a different choice yields better results. Be advised that the quality of the forecast primarily depends on the predictability of the data.

Save the Experiment

Once you are happy with the results of your Experiment, save it. Saving your Experiment will result in the following:

  1. Assistant settings saved as an Experiment knowledge object.
  2. (As applicable) Draft model updated to an Experiment model.
  3. (As applicable) Affiliated scheduled trainings and alerts update to synchronize with the search SPL and trigger conditions.

You can load a saved Experiment by clicking the Experiment name.

Deploy and manage the forecast

After you interpret ,validate, and refine the Experiment, deploy the forecast.

Within the Experiment framework:

To manage your Experiment, perform the following steps:

  1. From the MLTK navigation bar, choose Experiments. A list of your saved Experiments populates.
  2. Click the Manage button available under the Actions column.

The toolkit supports the following Experiment management options:

  1. Create Experiment-level alerts.
    Updating a saved Experiment can affect affiliated alerts. Re-validate your alerts once you complete the changes.
    For more information about alerts, see Getting started with alerts in the Splunk Enterprise Alerting Manual.
  2. Edit the title and description of the Experiment.
  3. (As applicable) Manage alerts for a single Experiment.
  4. (As applicable) Schedule a training job for an Experiment.
  5. Delete an Experiment.

Experiments are always stored under the user's namespace, meaning that changing sharing settings and permissions on Experiments is not supported.

Outside the Experiment framework:

  1. Within the Experiment click Open in Search to generate a New Search tab for this same dataset. This new search will open in a new browser tab, away from the Assistant.
    This search query uses all data, not just the training set. You can adjust the SPL directly and see results immediately. You can also save the query as a Report, Dashboard Panel or Alert.
  2. Click Show SPL to generate a new modal window/ overlay showing the search query you used to forecast. Copy the SPL to use in other aspects of your Splunk instance.

[[Category:experiment5

Last modified on 31 August, 2018
PREVIOUS
Detect Categorical Outliers Experiment workflow
  NEXT
Cluster Numeric Events Experiment workflow

This documentation applies to the following versions of Splunk® Machine Learning Toolkit: 3.4.0


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters