Savitzky-Golay Filter example

You can add custom algorithms to the MLTK app. This example adds a Savitzky-Golay signal processing from SciPy.

Because the SciPy savgol_filter is only a function, work is performed using the fit method which returns the transformed values. For more information see the SciPy documentation: https://docs.scipy.org/doc/scipy/

Add the Savitzky-Golay Filter algorithm

Follow these steps to add the Savitzky-Golay Filter algorithm.

1. Register the Savitzky-Golay Filter algorithm.
2. Create the Python file.
3. Define the class.
4. Define the init method.
5. Define the fit method.

Register the Savitzky-Golay Filter algorithm

Register the algorithm in algos.conf using one of the following methods.

Register the algorithm using the REST API

Use the following curl command to register using the REST API:

$ curl -k -u admin:<admin pass> https://localhost:8089/servicesNS/nobody/Splunk_ML_Toolkit/configs/conf-algos -d name="SavgolFilter"

Register the algorithm manually

Modify or create the algos.conf file located in $SPLUNK_HOME/etc/apps/Splunk_ML_Toolkit/local/ and add the following stanza to register your algorithm:

 [SavgolFilter]

When you register the algorithm with this method, you must restart Splunk Enterprise.

Create the Python file

Create the Python file in the algos folder. For this example, you create $SPLUNK_HOME/etc/apps/Splunk_ML_Toolkit/bin/algos/SavgolFilter.py.

import numpy as np
from scipy.signal import savgol_filter
 
from base import BaseAlgo
from util.param_util import convert_params
from util import df_util

Define the class

Inherit from the BaseAlgo. The class name is the name of the algorithm.

class SavgolFilter(BaseAlgo):
    """Use SciPy's savgol_filter to run a filter over fields."""

Define the init method

The __init__ method passes the options from the search to the algorithm. Because there isn't an estimator class like other examples, attach the params to the object (self) to use later.

    def __init__(self, options):
        # set parameters
        params = options.get('params', {})
        out_params = convert_params(
            params,
            ints=['window_length', 'polyorder', 'deriv']
        )

        # set defaults for parameters
        if 'window_length' in out_params:
            self.window_length = out_params['window_length']
        else:
            self.window_length = 5

        if 'polyorder' in out_params:
            self.polyorder = out_params['polyorder']
        else:
            self.polyorder = 2

        if 'deriv' in out_params:
            self.deriv = out_params['deriv']
        else:
            self.deriv = 0

Define the fit method

The fit method is where you compute the correlations. And then returns the DataFrame.

   def fit(self, df, options):
        X = df.copy()
        X, nans, columns = df_util.prepare_features(X, self.feature_variables)
 
	# Define a wrapper function
        def f(x):
            return savgol_filter(x, self.window_length, self.polyorder, self.deriv)
 
	# Apply that function along each column of X
        y_hat = np.apply_along_axis(f, 0, X)
 
        names = ['SG_%s' % col for col in columns]
        output_df = df_util.create_output_dataframe(y_hat, nans, names)
        df = df_util.merge_predictions(df, output_df)
 
        return df

End-to-end example

This Savitzky-Golay Filter example covers the following tasks:

• Using the BaseAlgo class
• Converting parameters
• Using the prepare_features utility
• Using an arbitrary function to transform data

import numpy as np
from scipy.signal import savgol_filter

from base import BaseAlgo
from util.param_util import convert_params
from util import df_util


class SavgolFilter(BaseAlgo):

    def __init__(self, options):
        # set parameters
        params = options.get('params', {})
        out_params = convert_params(
            params,
            ints=['window_length', 'polyorder', 'deriv']
        )

        # set defaults for parameters
        if 'window_length' in out_params:
            self.window_length = out_params['window_length']
        else:
            self.window_length = 5

        if 'polyorder' in out_params:
            self.polyorder = out_params['polyorder']
        else:
            self.polyorder = 2

        if 'deriv' in out_params:
            self.deriv = out_params['deriv']
        else:
            self.deriv = 0

    def fit(self, df, options):
        X = df.copy()
        X, nans, columns = df_util.prepare_features(X, self.feature_variables)

        def f(x):
            return savgol_filter(x, self.window_length, self.polyorder, self.deriv)

        y_hat = np.apply_along_axis(f, 0, X)

        names = ['SG_%s' % col for col in columns]
        output_df = df_util.create_output_dataframe(y_hat, nans, names)
        df = df_util.merge_predictions(df, output_df)

        return df