MLTK compatibility with upcoming releases of Splunk Enterprise

New releases of Splunk Enterprise and Splunk Cloud Platform are scheduled for June 2025. These releases are required to help customers be compliant with Federal Information Processing Standards (FIPS). See https://csrc.nist.gov/projects/cryptographic-module-validation-program

Further details of these changes can be found in the following Lantern article: Preparing to upgrade from 9.x to the upcoming release of Splunk Enterprise and Cloud Platform

Splunk anticipates these new releases will cause breaking changes to Splunk Machine Learning Toolkit (MLTK) users on versions lower than version 5.5.0. If you use MLTK version 5.4.2 or lower, review the following guidelines to prepare for the Splunk Enterprise and Splunk Cloud Platform releases.

How does this affect MLTK?

The MLTK user interface (UI) makes calls to search endpoints. For MLTK versions that are lower than version 5.5.0, the endpoints that are called from the UI are not FIPS compliant. This means that MLTK version 5.4.2 and lower are not compatible with the upcoming releases of Splunk Enterprise and Splunk Cloud Platform.

What do I need to do if I am using an MLTK version lower than 5.5.0?

To avoid any degradation in the product experience you can upgrade to MLTK version 5.5.0 before the scheduled Splunk Enterprise and Splunk Cloud Platform updates.

Although MLTK upgrades can be performed after the Splunk Enterprise and Splunk Cloud Platform updates, it is better to complete the MLTK upgrades prior, so as to avoid any impact. This is particularly important for Splunk Cloud Platform users, where fleet upgrades are automated, subject to your organization's maintenance window policy.

What happens if I don't upgrade?

If you do not upgrade, the MLTK UI will not function as expected in upcoming Splunk Enterprise and Splunk Cloud Platform releases. This means that you will lose access to any experiments you have created in MLTK and you will not be able to create any new experiments in MLTK.

In addition, searches that are created from the MLTK experiment UI will not be executable in future Splunk Enterprise and Splunk Cloud Platform releases.

Any scheduled searches that you have already created using MLTK, including any searches that use the ML-SPL commands of fit or apply, will continue to work as expected. It is only the MLTK UI that will see degradation.

Are there any other reasons to upgrade MLTK?

MLTK version 5.5.0 provides some enhancements to the DensityFunction algorithm, as well as addressing vulnerabilities, as outlined in the following Splunk Security Advisory. See Splunk Security Advisory SVD-2024-1102.

If you are not yet using MLTK 5.4.2, that version resolved several critical and high vulnerabilities as outlined in the following Splunk Security Advisory. See Splunk Security Advisory SVD-2024-0801.

The upcoming MLTK 5.6.0 release will also offer several enhancements, such as large-language model (LLM) integrations and ONNX model improvements.

As a best practice, upgrade to new versions of MLTK as soon as possible.

What do I need to do if I am on a lower version than 5.5.0?

If you are on MLTK version 5.4.2 you can simply upgrade to MLTK version 5.5.0.

If you are on a version lower than 5.4.2 you will need to complete a few model retraining and validation steps after upgrading.

Why do I need to retrain and validate models for upgrades of MLTK versions lower than 5.4.2?

Version 5.4.2 MLTK upgraded scikit-learn from version 0.24.2 to version 1.5.1. This introduced breaking changes to some algorithm APIs. If you have trained models that are built using certain algorithms then you will need to retrain those models. See Upgrade the Machine Learning Toolkit for full details of the affected algorithms.

If you use the ML-SPL API additional steps might also be required to validate any models trained with custom algorithms. See Adding a custom algorithm to the Splunk Machine Learning Toolkit in the ML-SPL API Guide for more information.

How do I retrain my models?

Once you upgrade to MLTK version 5.5.0 you must re-run any model training searches to retrain your models. For example, searches that contain the fit command.

You can use the following steps to identify and retrain any affected models in your Splunk environment:

Additional steps might be required if you are using custom algorithms.

1. Identify your existing MLTK models in Splunk. Models are stored as lookups in Splunk, with a .mlmodel suffix.
2. Identify if any of these models are trained using algorithms affected by the scikit-learn patch in MLTK 5.4.2 and make a note of these models. The full list of affected algorithms can be found in Upgrade the Splunk Machine Learning Toolkit .
3. For models that are affected, identify the searches that were used to train the models and make sure those searches are available to you following the upgrade:
   1. Searches that train an MLTK model will contain the fit command be in the following format:

      | fit <algo_name> <fields_and_algo_options> into <model_name>

   2. Make a note of the time ranges that were used when training the models previously.
4. Upgrade to MLTK version 5.5.0. For steps, see Upgrade the Machine Learning Toolkit.
5. Retrain your models by running the searches you identified in step 3 against the appropriate time ranges.
6. Confirm that your models have been retrained and are compatible with MLTK 5.5.0:
   1. Running the following search with your retrained models will identify if they have been retrained successfully or not:

      | summary <model_name>

   2. If unsuccessful, repeat step 5 and troubleshoot the job inspector.

What do I need to do if I am using MLTK 5.5.0 or higher?

If you are already using MLTK version 5.5.0 or higher you don't need to do anything. Your instance of MLTK is already compatible with future Splunk Enterprise or Splunk Cloud Platform releases.