splunklib.searchcommands

Design Notes

  1. Commands are constrained to this ABNF grammar:

    command       = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
command-name  = alpha *( alpha / digit )
option        = option-name [wsp] "=" [wsp] option-value
option-name   = alpha *( alpha / digit / "_" )
option-value  = word / quoted-string
word          = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
quoted-string = dquote *( word / wsp / "" dquote / dquote dquote ) dquote
field-name    = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )

    It does not show that field-name values may be comma-separated. This is because Splunk strips commas from the command line. A search command will never see them.

  2. Search commands targeting versions of Splunk prior to 6.3 must be statically configured as follows:

    1
2
3
4
    [command_name]
filename = command_name.py
supports_getinfo = true
supports_rawargs = true

    No other static configuration is required or expected and may interfere with command execution.

  3. Commands support dynamic probing for settings.

    Splunk probes for settings dynamically when supports_getinfo=true. You must add this line to the commands.conf stanza for each of your search commands.

  4. Commands do not support parsed arguments on the command line.

    Splunk parses arguments when supports_rawargs=false. The SearchCommand class sets this value unconditionally. You cannot override it.

    Rationale

    Splunk parses arguments by stripping quotes, nothing more. This may be useful in some cases, but doesn’t work well with our chosen grammar.

  5. Commands consume input headers.

    An input header is provided by Splunk when enableheader=true. The SearchCommand class sets this value unconditionally. You cannot override it.

  6. Commands produce an output messages header.

    Splunk expects a command to produce an output messages header when outputheader=true. The SearchCommand class sets this value unconditionally. You cannot override it.

  7. Commands support multi-value fields.

    Multi-value fields are provided and consumed by Splunk when supports_multivalue=true. This value is fixed. You cannot override it.

  8. This module represents all fields on the output stream in multi-value format.

    Splunk recognizes two kinds of data: value and list(value). The multi-value format represents these data in field pairs. Given field name the multi-value format calls for the creation of this pair of fields.

    Field name Field data
    name Value or text from which a list of values was derived.
    __mv_name Empty, if field represents a value; otherwise, an encoded list(value). Values in the list are wrapped in dollar signs ($) and separated by semi-colons (;). Dollar signs ($) within a value are represented by a pair of dollar signs ($$).

    Serializing data in this format enables streaming and reduces a command’s memory footprint at the cost of one extra byte of data per field per record and a small amount of extra processing time by the next command in the pipeline.

  9. A ReportingCommand must override reduce() and may override map(). Map/reduce commands on the Splunk processing pipeline are distinguished as this example illustrates.

    Splunk command

    sum total=total_date_hour date_hour

    Map command line

    sum __GETINFO__ __map__ total=total_date_hour date_hour
sum __EXECUTE__ __map__ total=total_date_hour date_hour

    Reduce command line

    sum __GETINFO__ total=total_date_hour date_hour
sum __EXECUTE__ total=total_date_hour date_hour

    The __map__ argument is introduced by ReportingCommand._execute(). Search command authors cannot influence the contents of the command line in this release.

References

  1. Custom Search Command manual:
  2. Create Custom Search Commands with commands.conf.spec
  3. Configure seach assistant with searchbnf.conf
  4. Control search distribution with distsearch.conf
splunklib.searchcommands.dispatch(command_class[, argv=sys.argv, input_file=sys.stdin, output_file=sys.stdout, module_name=None, allow_empty_input=True])

Instantiates and executes a search command class

This function implements a conditional script stanza based on the value of module_name:

if module_name is None or module_name == '__main__':
    # execute command

Call this function at module scope with module_name=__name__, if you would like your module to act as either a reusable module or a standalone program. Otherwise, if you wish this function to unconditionally instantiate and execute command_class, pass None as the value of module_name.

Parameters:
  • command_class (type) – Search command class to instantiate and execute.
  • argv (list or tuple) – List of arguments to the command.
  • input_file (file) – File from which the command will read data.
  • output_file (file) – File to which the command will write data.
  • module_name (basestring) – Name of the module calling dispatch or None.
  • allow_empty_input (bool) – Allow empty input records for the command, if False an Error will be returned if empty chunk body is encountered when read
Returns:

None

Example

1
2
3
4
5
6
7
8
#!/usr/bin/env python
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class SomeStreamingCommand(StreamingCommand):
    ...
    def stream(records):
        ...
dispatch(SomeStreamingCommand, module_name=__name__)

Dispatches the SomeStreamingCommand, if and only if __name__ is equal to '__main__'.

Example

1
2
3
4
5
6
7
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class SomeStreamingCommand(StreamingCommand):
    ...
    def stream(records):
        ...
dispatch(SomeStreamingCommand)

Unconditionally dispatches SomeStreamingCommand.

class splunklib.searchcommands.EventingCommand

Applies a transformation to search results as they travel through the events pipeline.

Eventing commands typically filter, group, order, and/or or augment event records. Examples of eventing commands from Splunk’s built-in command set include sort, dedup, and cluster. Each execution of an eventing command should produce a set of event records that is independently usable by downstream processors.

You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.

class ConfigurationSettings(command)

Represents the configuration settings that apply to a EventingCommand.

transform(records)

Generator function that processes and yields event records to the Splunk events pipeline.

You must override this method.

process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])

Process data.

Parameters:
  • argv (list or tuple) – Command line arguments.
  • ifile (file) – Input data file.
  • ofile (file) – Output data file.
  • allow_empty_input (bool) – Allow empty input records for the command, if False an Error will be returned if empty chunk body is encountered when read
Returns:

None
Return type:

NoneType
configuration

Returns the configuration settings for this command.

fieldnames

Returns the fieldnames specified as argument to this command.

finish()

Flushes the output buffer and signals that this command has finished processing data.

Returns:None
flush()

Flushes the output buffer.

Returns:None
input_header

Returns the input header for this command.

Returns:The input header for this command.
Return type:InputHeader
logger

Returns the logger for this command.

Returns:The logger for this command.
Return type:
logging_configuration

Syntax: logging_configuration=<path>

Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.

logging_level

Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]

Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.

options

Returns the options specified as argument to this command.

prepare()

Prepare for execution.

This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.

Returns:None
Return type:NoneType
search_results_info

Returns the search results info for this command invocation.

The search results info object is created from the search results info file associated with the command invocation.

Returns:Search results info:const:None, if the search results info file associated with the command invocation is inaccessible.
Return type:SearchResultsInfo or NoneType
service

Returns a Splunk service object for this command invocation or None.

The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:

enableheader = true
requires_srinfo = true

The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.

Returns:splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned.
write_metric(name, value)

Writes a metric that will be added to the search inspector.

Parameters:
  • name (basestring) – Name of the metric.
  • value –

    A 4-tuple containing the value of metric name where

    value[0] = Elapsed seconds or None. value[1] = Number of invocations or None. value[2] = Input count or None. value[3] = Output count or None.

The SearchMetric type provides a convenient encapsulation of value. The SearchMetric type provides a convenient encapsulation of value.

Returns:None.
class splunklib.searchcommands.GeneratingCommand

Generates events based on command arguments.

Generating commands receive no input and must be the first command on a pipeline. There are three pipelines: streams, events, and reports. The streams pipeline generates or processes time-ordered event records on an indexer or search head.

Streaming commands filter, modify, or augment event records and can be applied to subsets of index data in a parallel manner. An example of a streaming command from Splunk’s built-in command set is rex which extracts and adds fields to event records at search time. Records that pass through the streams pipeline move on to the events pipeline.

The events pipeline generates or processes records on a search head. Eventing commands typically filter, group, order, or augment event records. Examples of eventing commands from Splunk’s built-in command set include sort, dedup, and cluster. Each execution of an eventing command should produce a set of event records that is independently usable by downstream processors. Records that pass through the events pipeline move on to the reports pipeline.

The reports pipeline also runs on a search head, but yields data structures for presentation, not event records. Examples of streaming from Splunk’s built-in command set include chart, stats, and contingency.

Configure your generating command based on the pipeline that it targets. How you configure your command depends on the Search Command Protocol (SCP) version.

Pipeline SCP 1 SCP 2
streams streaming=True[,local=[True|False]] type=’streaming’[,distributed=[true|false]
events retainsevents=True, streaming=False type=’events’
reports streaming=False type=’reporting’

Only streaming commands may be distributed to indexers. By default generating commands are configured to run locally in the streams pipeline and will run under either SCP 1 or SCP 2.

@Configuration()
class StreamingGeneratingCommand(GeneratingCommand)
    ...

How you configure your command to run on a different pipeline or in a distributed fashion depends on what SCP protocol versions you wish to support. You must be sure to configure your command consistently for each protocol, if you wish to support both protocol versions correctly.

Commands configured like this will run as the first command on search heads and/or indexers on the streams pipeline.

Pipeline SCP 1 SCP 2
streams

  1. Add this line to your command’s stanza in

    default/commands.conf:

    local = false

  2. Restart splunk

  1. Add this configuration setting to your code:

    @Configuration(distributed=True)
class SomeCommand(GeneratingCommand)
    ...

  2. You are good to go; no need to restart Splunk

Generating commands configured like this will run as the first command on a search head on the events pipeline.

Pipeline SCP 1 SCP 2
events

You have a choice. Add these configuration settings to your command class:

@Configuration(
    retainsevents=True, streaming=False)
class SomeCommand(GeneratingCommand)
    ...

Or add these lines to default/commands.conf:

retainsevents = true
streaming = false

Add this configuration setting to your command setting to your command class:

@Configuration(type='events')
class SomeCommand(GeneratingCommand)
    ...

Configure your command class like this, if you wish to support both protocols:

@Configuration(type='events', retainsevents=True, streaming=False)
class SomeCommand(GeneratingCommand)
    ...

You might also consider adding these lines to commands.conf instead of adding them to your command class:

retainsevents = false
streaming = false

Commands configured like this will run as the first command on a search head on the reports pipeline.

Pipeline SCP 1 SCP 2
events

You have a choice. Add these configuration settings to your command class:

@Configuration(retainsevents=False)
class SomeCommand(GeneratingCommand)
    ...

Or add this lines to default/commands.conf:

retainsevents = false
streaming = false

Add this configuration setting to your command setting to your command class:

@Configuration(type='reporting')
class SomeCommand(GeneratingCommand)
    ...

Configure your command class like this, if you wish to support both protocols:

@Configuration(type='reporting', streaming=False)
class SomeCommand(GeneratingCommand)
    ...

You might also consider adding these lines to commands.conf instead of adding them to your command class:

retainsevents = false
streaming = false
class ConfigurationSettings(command)

Represents the configuration settings for a GeneratingCommand class.

generate()

A generator that yields records to the Splunk processing pipeline

You must override this method.

process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout, allow_empty_input=True])

Process data.

Parameters:
  • argv (list or tuple) – Command line arguments.
  • ifile (file) – Input data file.
  • ofile (file) – Output data file.
  • allow_empty_input (bool) – For generating commands, it must be true. Doing otherwise will cause an error.
Returns:

None
Return type:

NoneType
configuration

Returns the configuration settings for this command.

fieldnames

Returns the fieldnames specified as argument to this command.

finish()

Flushes the output buffer and signals that this command has finished processing data.

Returns:None
flush()

Flushes the output buffer.

Returns:None
input_header

Returns the input header for this command.

Returns:The input header for this command.
Return type:InputHeader
logger

Returns the logger for this command.

Returns:The logger for this command.
Return type:
logging_configuration

Syntax: logging_configuration=<path>

Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.

logging_level

Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]

Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.

options

Returns the options specified as argument to this command.

prepare()

Prepare for execution.

This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.

Returns:None
Return type:NoneType
search_results_info

Returns the search results info for this command invocation.

The search results info object is created from the search results info file associated with the command invocation.

Returns:Search results info:const:None, if the search results info file associated with the command invocation is inaccessible.
Return type:SearchResultsInfo or NoneType
service

Returns a Splunk service object for this command invocation or None.

The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:

enableheader = true
requires_srinfo = true

The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.

Returns:splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned.
write_metric(name, value)

Writes a metric that will be added to the search inspector.

Parameters:
  • name (basestring) – Name of the metric.
  • value –

    A 4-tuple containing the value of metric name where

    value[0] = Elapsed seconds or None. value[1] = Number of invocations or None. value[2] = Input count or None. value[3] = Output count or None.

The SearchMetric type provides a convenient encapsulation of value. The SearchMetric type provides a convenient encapsulation of value.

Returns:None.
class splunklib.searchcommands.ReportingCommand

Processes search result records and generates a reporting data structure.

Reporting search commands run as either reduce or map/reduce operations. The reduce part runs on a search head and is responsible for processing a single chunk of search results to produce the command’s reporting data structure. The map part is called a streaming preop. It feeds the reduce part with partial results and by default runs on the search head and/or one or more indexers.

You must implement a reduce() method as a generator function that iterates over a set of event records and yields a reporting data structure. You may implement a map() method as a generator function that iterates over a set of event records and yields dict or list(dict) instances.

Configure the map() operation using a Configuration decorator on your map() method. Configure it like you would a StreamingCommand. Configure the reduce() operation using a Configuration decorator on your ReportingCommand() class.

You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.

class ConfigurationSettings(command)

Represents the configuration settings for a ReportingCommand.

map(records)

Override this method to compute partial results.

Parameters:records –

You must override this method, if requires_preop=True.

process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])

Process data.

Parameters:
  • argv (list or tuple) – Command line arguments.
  • ifile (file) – Input data file.
  • ofile (file) – Output data file.
  • allow_empty_input (bool) – Allow empty input records for the command, if False an Error will be returned if empty chunk body is encountered when read
Returns:

None
Return type:

NoneType
reduce(records)

Override this method to produce a reporting data structure.

You must override this method.

configuration

Returns the configuration settings for this command.

fieldnames

Returns the fieldnames specified as argument to this command.

finish()

Flushes the output buffer and signals that this command has finished processing data.

Returns:None
flush()

Flushes the output buffer.

Returns:None
input_header

Returns the input header for this command.

Returns:The input header for this command.
Return type:InputHeader
logger

Returns the logger for this command.

Returns:The logger for this command.
Return type:
logging_configuration

Syntax: logging_configuration=<path>

Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.

logging_level

Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]

Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.

options

Returns the options specified as argument to this command.

prepare()

Prepare for execution.

This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.

Returns:None
Return type:NoneType
search_results_info

Returns the search results info for this command invocation.

The search results info object is created from the search results info file associated with the command invocation.

Returns:Search results info:const:None, if the search results info file associated with the command invocation is inaccessible.
Return type:SearchResultsInfo or NoneType
service

Returns a Splunk service object for this command invocation or None.

The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:

enableheader = true
requires_srinfo = true

The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.

Returns:splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned.
write_metric(name, value)

Writes a metric that will be added to the search inspector.

Parameters:
  • name (basestring) – Name of the metric.
  • value –

    A 4-tuple containing the value of metric name where

    value[0] = Elapsed seconds or None. value[1] = Number of invocations or None. value[2] = Input count or None. value[3] = Output count or None.

The SearchMetric type provides a convenient encapsulation of value. The SearchMetric type provides a convenient encapsulation of value.

Returns:None.
class splunklib.searchcommands.StreamingCommand

Applies a transformation to search results as they travel through the streams pipeline.

Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records. Hence, a search command must be prepared to be invoked many times during the course of pipeline processing. Each invocation should produce a set of results independently usable by downstream processors.

By default Splunk may choose to run a streaming command locally on a search head and/or remotely on one or more indexers concurrently. The size and frequency of the search result batches sent to the command will vary based on scheduling considerations.

You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.

class ConfigurationSettings(command)

Represents the configuration settings that apply to a StreamingCommand.

process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout, allow_empty_input=True])

Process data.

Parameters:
  • argv (list or tuple) – Command line arguments.
  • ifile (file) – Input data file.
  • ofile (file) – Output data file.
  • allow_empty_input (bool) – Allow empty input records for the command, if False an Error will be returned if empty chunk body is encountered when read
Returns:

None
Return type:

NoneType
stream(records)

Generator function that processes and yields event records to the Splunk stream pipeline.

You must override this method.

configuration

Returns the configuration settings for this command.

fieldnames

Returns the fieldnames specified as argument to this command.

finish()

Flushes the output buffer and signals that this command has finished processing data.

Returns:None
flush()

Flushes the output buffer.

Returns:None
input_header

Returns the input header for this command.

Returns:The input header for this command.
Return type:InputHeader
logger

Returns the logger for this command.

Returns:The logger for this command.
Return type:
logging_configuration

Syntax: logging_configuration=<path>

Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.

logging_level

Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]

Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.

options

Returns the options specified as argument to this command.

prepare()

Prepare for execution.

This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.

Returns:None
Return type:NoneType
search_results_info

Returns the search results info for this command invocation.

The search results info object is created from the search results info file associated with the command invocation.

Returns:Search results info:const:None, if the search results info file associated with the command invocation is inaccessible.
Return type:SearchResultsInfo or NoneType
service

Returns a Splunk service object for this command invocation or None.

The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:

enableheader = true
requires_srinfo = true

The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.

Returns:splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned.
write_metric(name, value)

Writes a metric that will be added to the search inspector.

Parameters:
  • name (basestring) – Name of the metric.
  • value –

    A 4-tuple containing the value of metric name where

    value[0] = Elapsed seconds or None. value[1] = Number of invocations or None. value[2] = Input count or None. value[3] = Output count or None.

The SearchMetric type provides a convenient encapsulation of value. The SearchMetric type provides a convenient encapsulation of value.

Returns:None.
class splunklib.searchcommands.Configuration(o=None, **kwargs)

Defines the configuration settings for a search command.

Documents, validates, and ensures that only relevant configuration settings are applied. Adds a name class variable to search command classes that don’t have one. The name is derived from the name of the class. By convention command class names end with the word “Command”. To derive name the word “Command” is removed from the end of the class name and then converted to lower case for conformance with the Search command style guide

class splunklib.searchcommands.Option(fget=None, fset=None, fdel=None, doc=None, name=None, default=None, require=None, validate=None)

Represents a search command option.

Required options must be specified on the search command line.

Example:

Short form (recommended). When you are satisfied with built-in or custom validation behaviors.

1
2
3
4
5
6
7
8
from splunklib.searchcommands.decorators import Option
from splunklib.searchcommands.validators import Fieldname

total = Option(
    doc=''' **Syntax:** **total=***<fieldname>*
    **Description:** Name of the field that will hold the computed
    sum''',
    require=True, validate=Fieldname())

Example:

Long form. Useful when you wish to manage the option value and its deleter/getter/setter side-effects yourself. You must provide a getter and a setter. If your Option requires destruction you must also provide a deleter. You must be prepared to accept a value of None which indicates that your Option is unset.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
from splunklib.searchcommands import Option

@Option()
def logging_configuration(self):
    """ **Syntax:** logging_configuration=<path>
    **Description:** Loads an alternative logging configuration file for a command invocation. The logging
    configuration file must be in Python ConfigParser-format. The *<path>* name and all path names specified in
    configuration are relative to the app root directory.

    """
    return self._logging_configuration

@logging_configuration.setter
def logging_configuration(self, value):
    if value is not None
        logging.configure(value)
        self._logging_configuration = value

def __init__(self)
    self._logging_configuration = None
deleter(function)

Descriptor to change the deleter on a property.

getter(function)

Descriptor to change the getter on a property.

setter(function)

Descriptor to change the setter on a property.

class splunklib.searchcommands.Boolean

Validates Boolean option values.

class splunklib.searchcommands.Duration

Validates duration option values.

class splunklib.searchcommands.File(mode='rt', buffering=None, directory=None)

Validates file option values.

class splunklib.searchcommands.Integer(minimum=None, maximum=None)

Validates integer option values.

class splunklib.searchcommands.Float(minimum=None, maximum=None)

Validates float option values.

class splunklib.searchcommands.RegularExpression

Validates regular expression option values.

class splunklib.searchcommands.Set(*args)

Validates set option values.