splunklib.searchcommands¶

Design Notes

Commands are constrained to this ABNF grammar:

command       = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
command-name  = alpha *( alpha / digit )
option        = option-name [wsp] "=" [wsp] option-value
option-name   = alpha *( alpha / digit / "_" )
option-value  = word / quoted-string
word          = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
quoted-string = dquote *( word / wsp / "" dquote / dquote dquote ) dquote
field-name    = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )

It does not show that field-name values may be comma-separated. This is because Splunk strips commas from

the command line. A search command will never see them.

Search commands targeting versions of Splunk prior to 6.3 must be statically configured as follows:
1 2 3 4
[command_name] filename = command_name.py supports_getinfo = true supports_rawargs = true
No other static configuration is required or expected and may interfere with command execution.
Commands support dynamic probing for settings.

Splunk probes for settings dynamically when supports_getinfo=true. You must add this line to the commands.conf stanza for each of your search commands.
Commands do not support parsed arguments on the command line.

Splunk parses arguments when supports_rawargs=false. The SearchCommand class sets this value unconditionally. You cannot override it.

Rationale

Splunk parses arguments by stripping quotes, nothing more. This may be useful in some cases, but doesn’t work well with our chosen grammar.
Commands consume input headers.

An input header is provided by Splunk when enableheader=true. The SearchCommand class sets this value unconditionally. You cannot override it.
Commands produce an output messages header.

Splunk expects a command to produce an output messages header when outputheader=true. The SearchCommand class sets this value unconditionally. You cannot override it.
Commands support multi-value fields.

Multi-value fields are provided and consumed by Splunk when supports_multivalue=true. This value is fixed. You cannot override it.

This module represents all fields on the output stream in multi-value format.

Splunk recognizes two kinds of data: value and list(value). The multi-value format represents these data in field pairs. Given field name the multi-value format calls for the creation of this pair of fields.

Field name	Field data
`name`	Value or text from which a list of values was derived.
`__mv_name`	Empty, if `field` represents a `value`; otherwise, an encoded `list(value)`. Values in the list are wrapped in dollar signs ($) and separated by semi-colons (;). Dollar signs ($) within a value are represented by a pair of dollar signs ($$).

Serializing data in this format enables streaming and reduces a command’s memory footprint at the cost of one extra byte of data per field per record and a small amount of extra processing time by the next command in the pipeline.

A ReportingCommand must override reduce() and may override map(). Map/reduce commands on the Splunk processing pipeline are distinguished as this example illustrates.

Splunk command
```
sum total=total_date_hour date_hour
```
Map command line
```
sum __GETINFO__ __map__ total=total_date_hour date_hour
sum __EXECUTE__ __map__ total=total_date_hour date_hour
```
Reduce command line
```
sum __GETINFO__ total=total_date_hour date_hour
sum __EXECUTE__ total=total_date_hour date_hour
```
The __map__ argument is introduced by ReportingCommand._execute(). Search command authors cannot influence the contents of the command line in this release.

References

splunklib.searchcommands.dispatch(command_class[, argv=sys.argv, input_file=sys.stdin, output_file=sys.stdout, module_name=None])¶

Instantiates and executes a search command class

This function implements a conditional script stanza based on the value of module_name:

if module_name is None or module_name == '__main__':
    # execute command

Call this function at module scope with module_name=__name__, if you would like your module to act as either a reusable module or a standalone program. Otherwise, if you wish this function to unconditionally instantiate and execute command_class, pass None as the value of module_name.

Parameters:	command_class (type) – Search command class to instantiate and execute. argv (list or tuple) – List of arguments to the command. input_file (`file`) – File from which the command will read data. output_file (`file`) – File to which the command will write data. module_name (`basestring`) – Name of the module calling `dispatch` or `None`.
Returns:	`None`

Example

#!/usr/bin/env python
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class SomeStreamingCommand(StreamingCommand):
    ...
    def stream(records):
        ...
dispatch(SomeStreamingCommand, module_name=__name__)

Dispatches the SomeStreamingCommand, if and only if __name__ is equal to '__main__'.

Example

from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class SomeStreamingCommand(StreamingCommand):
    ...
    def stream(records):
        ...
dispatch(SomeStreamingCommand)

Unconditionally dispatches SomeStreamingCommand.

class splunklib.searchcommands.EventingCommand¶

Applies a transformation to search results as they travel through the events pipeline.

Eventing commands typically filter, group, order, and/or or augment event records. Examples of eventing commands from Splunk’s built-in command set include sort, dedup, and cluster. Each execution of an eventing command should produce a set of event records that is independently usable by downstream processors.

You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.

class ConfigurationSettings(command)¶: Represents the configuration settings that apply to a EventingCommand.

EventingCommand.transform(records)¶

Generator function that processes and yields event records to the Splunk events pipeline.

You must override this method.

EventingCommand.process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])¶

Process data.

Parameters:	argv (list or tuple) – Command line arguments. ifile (file) – Input data file. ofile (file) – Output data file.
Returns:	`None`
Return type:	NoneType

EventingCommand.configuration¶: Returns the configuration settings for this command.

EventingCommand.fieldnames¶: Returns the fieldnames specified as argument to this command.

EventingCommand.finish()¶

Flushes the output buffer and signals that this command has finished processing data.

Returns:	`None`

EventingCommand.flush()¶

Flushes the output buffer.

Returns:	`None`

EventingCommand.input_header¶

Returns the input header for this command.

Returns:	The input header for this command.
Return type:	InputHeader

EventingCommand.logger¶

Returns the logger for this command.

Returns:	The logger for this command.
Return type:

EventingCommand.logging_configuration¶

Syntax: logging_configuration=<path>

Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.

EventingCommand.logging_level¶

Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.

EventingCommand.options¶: Returns the options specified as argument to this command.

EventingCommand.prepare()¶

Prepare for execution.

This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.

Returns:	`None`
Return type:	NoneType

EventingCommand.search_results_info¶

Returns the search results info for this command invocation.

The search results info object is created from the search results info file associated with the command invocation.

Returns:	Search results info:const:None, if the search results info file associated with the command

invocation is inaccessible. :rtype: SearchResultsInfo or NoneType

EventingCommand.service¶

Returns a Splunk service object for this command invocation or None.

The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:

The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.

Returns:	`splunklib.client.Service`, if `enableheader` and `requires_srinfo` are both

true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned.

EventingCommand.write_metric(name, value)¶

Writes a metric that will be added to the search inspector.

Parameters:	name (basestring) – Name of the metric. value – A 4-tuple containing the value of metric :param:`name` where value[0] = Elapsed seconds or `None`. value[1] = Number of invocations or `None`. value[2] = Input count or `None`. value[3] = Output count or `None`.

The SearchMetric type provides a convenient encapsulation of :param:`value`. The SearchMetric type provides a convenient encapsulation of :param:`value`.

Returns:	`None`.

class splunklib.searchcommands.GeneratingCommand¶

Generates events based on command arguments.

Generating commands receive no input and must be the first command on a pipeline. There are three pipelines: streams, events, and reports. The streams pipeline generates or processes time-ordered event records on an indexer or search head.

Streaming commands filter, modify, or augment event records and can be applied to subsets of index data in a parallel manner. An example of a streaming command from Splunk’s built-in command set is rex which extracts and adds fields to event records at search time. Records that pass through the streams pipeline move on to the events pipeline.

The events pipeline generates or processes records on a search head. Eventing commands typically filter, group, order, or augment event records. Examples of eventing commands from Splunk’s built-in command set include sort, dedup, and cluster. Each execution of an eventing command should produce a set of event records that is independently usable by downstream processors. Records that pass through the events pipeline move on to the reports pipeline.

The reports pipeline also runs on a search head, but yields data structures for presentation, not event records. Examples of streaming from Splunk’s built-in command set include chart, stats, and contingency.

Configure your generating command based on the pipeline that it targets. How you configure your command depends on the Search Command Protocol (SCP) version.

Pipeline	SCP 1	SCP 2
streams	streaming=True[,local=[True\|False]]	type=’streaming’[,distributed=[true\|false]
events	retainsevents=True, streaming=False	type=’eventing’
reports	streaming=False	type=’reporting’

Only streaming commands may be distributed to indexers. By default generating commands are configured to run locally in the streams pipeline and will run under either SCP 1 or SCP 2.

@Configuration()
class StreamingGeneratingCommand(GeneratingCommand)
    ...

How you configure your command to run on a different pipeline or in a distributed fashion depends on what SCP protocol versions you wish to support. You must be sure to configure your command consistently for each protocol, if you wish to support both protocol versions correctly.

Commands configured like this will run as the first command on search heads and/or indexers on the streams pipeline.

Pipeline	SCP 1	SCP 2
streams	Add this line to your command’s stanza in default/commands.conf. .. code-block:: python local = false Restart splunk	Add this configuration setting to your code: You are good to go; no need to restart Splunk

Generating commands configured like this will run as the first command on a search head on the events pipeline.

Pipeline

SCP 1

SCP 2

events

You have a choice. Add these configuration settings to your command class:

Or add these lines to default/commands.conf:

Add this configuration setting to your command setting to your command class:

Configure your command class like this, if you wish to support both protocols:

You might also consider adding these lines to commands.conf instead of adding them to your command class:

Commands configured like this will run as the first command on a search head on the reports pipeline.

Pipeline

SCP 1

SCP 2

events

You have a choice. Add these configuration settings to your command class:

Or add this lines to default/commands.conf:

Add this configuration setting to your command setting to your command class:

Configure your command class like this, if you wish to support both protocols:

You might also consider adding these lines to commands.conf instead of adding them to your command class:

class ConfigurationSettings(command)¶: Represents the configuration settings for a GeneratingCommand class.

GeneratingCommand.generate()¶

A generator that yields records to the Splunk processing pipeline

You must override this method.

GeneratingCommand.process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])¶

Process data.

Parameters:	argv (list or tuple) – Command line arguments. ifile (file) – Input data file. ofile (file) – Output data file.
Returns:	`None`
Return type:	NoneType

GeneratingCommand.configuration¶: Returns the configuration settings for this command.

GeneratingCommand.fieldnames¶: Returns the fieldnames specified as argument to this command.

GeneratingCommand.finish()¶

Flushes the output buffer and signals that this command has finished processing data.

Returns:	`None`

GeneratingCommand.flush()¶

Flushes the output buffer.

Returns:	`None`

GeneratingCommand.input_header¶

Returns the input header for this command.

Returns:	The input header for this command.
Return type:	InputHeader

GeneratingCommand.logger¶

Returns the logger for this command.

Returns:	The logger for this command.
Return type:

GeneratingCommand.logging_configuration¶

Syntax: logging_configuration=<path>

Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.

GeneratingCommand.logging_level¶

Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.

GeneratingCommand.options¶: Returns the options specified as argument to this command.

GeneratingCommand.prepare()¶

Prepare for execution.

This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.

Returns:	`None`
Return type:	NoneType

GeneratingCommand.search_results_info¶

Returns the search results info for this command invocation.

The search results info object is created from the search results info file associated with the command invocation.

Returns:	Search results info:const:None, if the search results info file associated with the command

invocation is inaccessible. :rtype: SearchResultsInfo or NoneType

GeneratingCommand.service¶

Returns a Splunk service object for this command invocation or None.

The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:

The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.

Returns:	`splunklib.client.Service`, if `enableheader` and `requires_srinfo` are both

true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned.

GeneratingCommand.write_metric(name, value)¶

Writes a metric that will be added to the search inspector.

Parameters:	name (basestring) – Name of the metric. value – A 4-tuple containing the value of metric :param:`name` where value[0] = Elapsed seconds or `None`. value[1] = Number of invocations or `None`. value[2] = Input count or `None`. value[3] = Output count or `None`.

The SearchMetric type provides a convenient encapsulation of :param:`value`. The SearchMetric type provides a convenient encapsulation of :param:`value`.

Returns:	`None`.

class splunklib.searchcommands.ReportingCommand¶

Processes search result records and generates a reporting data structure.

Reporting search commands run as either reduce or map/reduce operations. The reduce part runs on a search head and is responsible for processing a single chunk of search results to produce the command’s reporting data structure. The map part is called a streaming preop. It feeds the reduce part with partial results and by default runs on the search head and/or one or more indexers.

You must implement a reduce() method as a generator function that iterates over a set of event records and yields a reporting data structure. You may implement a map() method as a generator function that iterates over a set of event records and yields dict or list(dict) instances.

Configure the map() operation using a Configuration decorator on your map() method. Configure it like you would a StreamingCommand. Configure the reduce() operation using a Configuration decorator on your ReportingCommand() class.

You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.

class ConfigurationSettings(command)¶: Represents the configuration settings for a ReportingCommand.

ReportingCommand.map(records)¶

Override this method to compute partial results.

Parameters:	records –

You must override this method, if requires_preop=True.

ReportingCommand.process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])¶

Process data.

Parameters:	argv (list or tuple) – Command line arguments. ifile (file) – Input data file. ofile (file) – Output data file.
Returns:	`None`
Return type:	NoneType

ReportingCommand.reduce(records)¶

Override this method to produce a reporting data structure.

You must override this method.

ReportingCommand.configuration¶: Returns the configuration settings for this command.

ReportingCommand.fieldnames¶: Returns the fieldnames specified as argument to this command.

ReportingCommand.finish()¶

Flushes the output buffer and signals that this command has finished processing data.

Returns:	`None`

ReportingCommand.flush()¶

Flushes the output buffer.

Returns:	`None`

ReportingCommand.input_header¶

Returns the input header for this command.

Returns:	The input header for this command.
Return type:	InputHeader

ReportingCommand.logger¶

Returns the logger for this command.

Returns:	The logger for this command.
Return type:

ReportingCommand.logging_configuration¶

Syntax: logging_configuration=<path>

Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.

ReportingCommand.logging_level¶

Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.

ReportingCommand.options¶: Returns the options specified as argument to this command.

ReportingCommand.search_results_info¶

Returns the search results info for this command invocation.

The search results info object is created from the search results info file associated with the command invocation.

Returns:	Search results info:const:None, if the search results info file associated with the command

invocation is inaccessible. :rtype: SearchResultsInfo or NoneType

ReportingCommand.service¶

Returns a Splunk service object for this command invocation or None.

The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:

The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.

Returns:	`splunklib.client.Service`, if `enableheader` and `requires_srinfo` are both

true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned.

ReportingCommand.write_metric(name, value)¶

Writes a metric that will be added to the search inspector.

Parameters:	name (basestring) – Name of the metric. value – A 4-tuple containing the value of metric :param:`name` where value[0] = Elapsed seconds or `None`. value[1] = Number of invocations or `None`. value[2] = Input count or `None`. value[3] = Output count or `None`.

The SearchMetric type provides a convenient encapsulation of :param:`value`. The SearchMetric type provides a convenient encapsulation of :param:`value`.

Returns:	`None`.

class splunklib.searchcommands.StreamingCommand¶

Applies a transformation to search results as they travel through the streams pipeline.

Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records. Hence, a search command must be prepared to be invoked many times during the course of pipeline processing. Each invocation should produce a set of results independently usable by downstream processors.

By default Splunk may choose to run a streaming command locally on a search head and/or remotely on one or more indexers concurrently. The size and frequency of the search result batches sent to the command will vary based on scheduling considerations.

You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.

class ConfigurationSettings(command)¶: Represents the configuration settings that apply to a StreamingCommand.

StreamingCommand.process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])¶

Process data.

Parameters:	argv (list or tuple) – Command line arguments. ifile (file) – Input data file. ofile (file) – Output data file.
Returns:	`None`
Return type:	NoneType

StreamingCommand.stream(records)¶

Generator function that processes and yields event records to the Splunk stream pipeline.

You must override this method.

StreamingCommand.configuration¶: Returns the configuration settings for this command.

StreamingCommand.fieldnames¶: Returns the fieldnames specified as argument to this command.

StreamingCommand.finish()¶

Flushes the output buffer and signals that this command has finished processing data.

Returns:	`None`

StreamingCommand.flush()¶

Flushes the output buffer.

Returns:	`None`

StreamingCommand.input_header¶

Returns the input header for this command.

Returns:	The input header for this command.
Return type:	InputHeader

StreamingCommand.logger¶

Returns the logger for this command.

Returns:	The logger for this command.
Return type:

StreamingCommand.logging_configuration¶

Syntax: logging_configuration=<path>

Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.

StreamingCommand.logging_level¶

Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.

StreamingCommand.options¶: Returns the options specified as argument to this command.

StreamingCommand.prepare()¶

Prepare for execution.

This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.

Returns:	`None`
Return type:	NoneType

StreamingCommand.search_results_info¶

Returns the search results info for this command invocation.

The search results info object is created from the search results info file associated with the command invocation.

Returns:	Search results info:const:None, if the search results info file associated with the command

invocation is inaccessible. :rtype: SearchResultsInfo or NoneType

StreamingCommand.service¶

Returns a Splunk service object for this command invocation or None.

The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:

The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.

Returns:	`splunklib.client.Service`, if `enableheader` and `requires_srinfo` are both

true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned.

StreamingCommand.write_metric(name, value)¶

Writes a metric that will be added to the search inspector.

Parameters:	name (basestring) – Name of the metric. value – A 4-tuple containing the value of metric :param:`name` where value[0] = Elapsed seconds or `None`. value[1] = Number of invocations or `None`. value[2] = Input count or `None`. value[3] = Output count or `None`.

The SearchMetric type provides a convenient encapsulation of :param:`value`. The SearchMetric type provides a convenient encapsulation of :param:`value`.

Returns:	`None`.

class splunklib.searchcommands.Configuration(o=None, **kwargs)¶

Defines the configuration settings for a search command.

Documents, validates, and ensures that only relevant configuration settings are applied. Adds a name class variable to search command classes that don’t have one. The name is derived from the name of the class. By convention command class names end with the word “Command”. To derive name the word “Command” is removed from the end of the class name and then converted to lower case for conformance with the Search command style guide

class splunklib.searchcommands.Option(fget=None, fset=None, fdel=None, doc=None, name=None, default=None, require=None, validate=None)¶

Represents a search command option.

Required options must be specified on the search command line.

Example:

Short form (recommended). When you are satisfied with built-in or custom validation behaviors.

Example:

Long form. Useful when you wish to manage the option value and its deleter/getter/setter side-effects yourself. You must provide a getter and a setter. If your Option requires destruction you must also provide a deleter. You must be prepared to accept a value of None which indicates that your Option is unset.

class splunklib.searchcommands.Boolean¶: Validates Boolean option values.

class splunklib.searchcommands.Duration¶: Validates duration option values.

class splunklib.searchcommands.File(mode=u'rt', buffering=None, directory=None)¶: Validates file option values.

class splunklib.searchcommands.Integer(minimum=None, maximum=None)¶: Validates integer option values.

class splunklib.searchcommands.RegularExpression¶: Validates regular expression option values.

class splunklib.searchcommands.Set(*args)¶: Validates set option values.

Navigation

splunklib.searchcommands¶

Table Of Contents

Quick search

Navigation