Design Notes
Commands are constrained to this ABNF grammar:
command = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
command-name = alpha *( alpha / digit )
option = option-name [wsp] "=" [wsp] option-value
option-name = alpha *( alpha / digit / "_" )
option-value = word / quoted-string
word = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
quoted-string = dquote *( word / wsp / "" dquote / dquote dquote ) dquote
field-name = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )
It does not show that field-name values may be comma-separated. This is because Splunk strips commas from the command line. A search command will never see them.
Search commands targeting versions of Splunk prior to 6.3 must be statically configured as follows:
1 2 3 4 | [command_name]
filename = command_name.py
supports_getinfo = true
supports_rawargs = true
|
No other static configuration is required or expected and may interfere with command execution.
Commands support dynamic probing for settings.
Splunk probes for settings dynamically when supports_getinfo=true. You must add this line to the commands.conf stanza for each of your search commands.
Commands do not support parsed arguments on the command line.
Splunk parses arguments when supports_rawargs=false. The SearchCommand class sets this value unconditionally. You cannot override it.
Rationale
Splunk parses arguments by stripping quotes, nothing more. This may be useful in some cases, but doesn’t work well with our chosen grammar.
Commands consume input headers.
An input header is provided by Splunk when enableheader=true. The SearchCommand class sets this value unconditionally. You cannot override it.
Commands produce an output messages header.
Splunk expects a command to produce an output messages header when outputheader=true. The SearchCommand class sets this value unconditionally. You cannot override it.
Commands support multi-value fields.
Multi-value fields are provided and consumed by Splunk when supports_multivalue=true. This value is fixed. You cannot override it.
This module represents all fields on the output stream in multi-value format.
Splunk recognizes two kinds of data: value and list(value). The multi-value format represents these data in field pairs. Given field name the multi-value format calls for the creation of this pair of fields.
Field name | Field data |
---|---|
name | Value or text from which a list of values was derived. |
__mv_name | Empty, if field represents a value; otherwise, an encoded list(value). Values in the list are wrapped in dollar signs ($) and separated by semi-colons (;). Dollar signs ($) within a value are represented by a pair of dollar signs ($$). |
Serializing data in this format enables streaming and reduces a command’s memory footprint at the cost of one extra byte of data per field per record and a small amount of extra processing time by the next command in the pipeline.
A ReportingCommand must override reduce() and may override map(). Map/reduce commands on the Splunk processing pipeline are distinguished as this example illustrates.
Splunk command
sum total=total_date_hour date_hour
Map command line
sum __GETINFO__ __map__ total=total_date_hour date_hour
sum __EXECUTE__ __map__ total=total_date_hour date_hour
Reduce command line
sum __GETINFO__ total=total_date_hour date_hour
sum __EXECUTE__ total=total_date_hour date_hour
The __map__ argument is introduced by ReportingCommand._execute(). Search command authors cannot influence the contents of the command line in this release.
References
Instantiates and executes a search command class
This function implements a conditional script stanza based on the value of module_name:
if module_name is None or module_name == '__main__':
# execute command
Call this function at module scope with module_name=__name__, if you would like your module to act as either a reusable module or a standalone program. Otherwise, if you wish this function to unconditionally instantiate and execute command_class, pass None as the value of module_name.
Parameters: |
|
---|---|
Returns: | None |
Example
1 2 3 4 5 6 7 8 | #!/usr/bin/env python
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class SomeStreamingCommand(StreamingCommand):
...
def stream(records):
...
dispatch(SomeStreamingCommand, module_name=__name__)
|
Dispatches the SomeStreamingCommand, if and only if __name__ is equal to '__main__'.
Example
1 2 3 4 5 6 7 | from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class SomeStreamingCommand(StreamingCommand):
...
def stream(records):
...
dispatch(SomeStreamingCommand)
|
Unconditionally dispatches SomeStreamingCommand.
Applies a transformation to search results as they travel through the events pipeline.
Eventing commands typically filter, group, order, and/or or augment event records. Examples of eventing commands from Splunk’s built-in command set include sort, dedup, and cluster. Each execution of an eventing command should produce a set of event records that is independently usable by downstream processors.
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.
Represents the configuration settings that apply to a EventingCommand.
Generator function that processes and yields event records to the Splunk events pipeline.
You must override this method.
Process data.
Parameters: |
|
---|---|
Returns: | None |
Return type: | NoneType |
Returns the configuration settings for this command.
Returns the fieldnames specified as argument to this command.
Flushes the output buffer and signals that this command has finished processing data.
Returns: | None |
---|
Flushes the output buffer.
Returns: | None |
---|
Returns the input header for this command.
Returns: | The input header for this command. |
---|---|
Return type: | InputHeader |
Returns the logger for this command.
Returns: | The logger for this command. |
---|---|
Return type: |
Syntax: logging_configuration=<path>
Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.
Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]
Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.
Returns the options specified as argument to this command.
Prepare for execution.
This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.
Returns: | None |
---|---|
Return type: | NoneType |
Returns the search results info for this command invocation.
The search results info object is created from the search results info file associated with the command invocation.
Returns: | Search results info:const:None, if the search results info file associated with the command invocation is inaccessible. |
---|---|
Return type: | SearchResultsInfo or NoneType |
Returns a Splunk service object for this command invocation or None.
The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:
enableheader = true requires_srinfo = true
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
Returns: | splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
---|
Writes a metric that will be added to the search inspector.
Parameters: |
|
---|
The SearchMetric type provides a convenient encapsulation of value. The SearchMetric type provides a convenient encapsulation of value.
Returns: | None. |
---|
Generates events based on command arguments.
Generating commands receive no input and must be the first command on a pipeline. There are three pipelines: streams, events, and reports. The streams pipeline generates or processes time-ordered event records on an indexer or search head.
Streaming commands filter, modify, or augment event records and can be applied to subsets of index data in a parallel manner. An example of a streaming command from Splunk’s built-in command set is rex which extracts and adds fields to event records at search time. Records that pass through the streams pipeline move on to the events pipeline.
The events pipeline generates or processes records on a search head. Eventing commands typically filter, group, order, or augment event records. Examples of eventing commands from Splunk’s built-in command set include sort, dedup, and cluster. Each execution of an eventing command should produce a set of event records that is independently usable by downstream processors. Records that pass through the events pipeline move on to the reports pipeline.
The reports pipeline also runs on a search head, but yields data structures for presentation, not event records. Examples of streaming from Splunk’s built-in command set include chart, stats, and contingency.
Configure your generating command based on the pipeline that it targets. How you configure your command depends on the Search Command Protocol (SCP) version.
Pipeline | SCP 1 | SCP 2 |
---|---|---|
streams | streaming=True[,local=[True|False]] | type=’streaming’[,distributed=[true|false] |
events | retainsevents=True, streaming=False | type=’events’ |
reports | streaming=False | type=’reporting’ |
Only streaming commands may be distributed to indexers. By default generating commands are configured to run locally in the streams pipeline and will run under either SCP 1 or SCP 2.
@Configuration()
class StreamingGeneratingCommand(GeneratingCommand)
...
How you configure your command to run on a different pipeline or in a distributed fashion depends on what SCP protocol versions you wish to support. You must be sure to configure your command consistently for each protocol, if you wish to support both protocol versions correctly.
Commands configured like this will run as the first command on search heads and/or indexers on the streams pipeline.
Pipeline | SCP 1 | SCP 2 |
---|---|---|
streams |
|
|
Generating commands configured like this will run as the first command on a search head on the events pipeline.
Pipeline | SCP 1 | SCP 2 |
---|---|---|
events | You have a choice. Add these configuration settings to your command class: @Configuration(
retainsevents=True, streaming=False)
class SomeCommand(GeneratingCommand)
...
Or add these lines to default/commands.conf: retainsevents = true
streaming = false
|
Add this configuration setting to your command setting to your command class: @Configuration(type='events')
class SomeCommand(GeneratingCommand)
...
|
Configure your command class like this, if you wish to support both protocols:
@Configuration(type='events', retainsevents=True, streaming=False)
class SomeCommand(GeneratingCommand)
...
You might also consider adding these lines to commands.conf instead of adding them to your command class:
retainsevents = false
streaming = false
Commands configured like this will run as the first command on a search head on the reports pipeline.
Pipeline | SCP 1 | SCP 2 |
---|---|---|
events | You have a choice. Add these configuration settings to your command class: @Configuration(retainsevents=False)
class SomeCommand(GeneratingCommand)
...
Or add this lines to default/commands.conf: retainsevents = false
streaming = false
|
Add this configuration setting to your command setting to your command class: @Configuration(type='reporting')
class SomeCommand(GeneratingCommand)
...
|
Configure your command class like this, if you wish to support both protocols:
@Configuration(type='reporting', streaming=False)
class SomeCommand(GeneratingCommand)
...
You might also consider adding these lines to commands.conf instead of adding them to your command class:
retainsevents = false
streaming = false
Represents the configuration settings for a GeneratingCommand class.
A generator that yields records to the Splunk processing pipeline
You must override this method.
Process data.
Parameters: |
|
---|---|
Returns: | None |
Return type: | NoneType |
Returns the configuration settings for this command.
Returns the fieldnames specified as argument to this command.
Flushes the output buffer and signals that this command has finished processing data.
Returns: | None |
---|
Flushes the output buffer.
Returns: | None |
---|
Returns the input header for this command.
Returns: | The input header for this command. |
---|---|
Return type: | InputHeader |
Returns the logger for this command.
Returns: | The logger for this command. |
---|---|
Return type: |
Syntax: logging_configuration=<path>
Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.
Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]
Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.
Returns the options specified as argument to this command.
Prepare for execution.
This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.
Returns: | None |
---|---|
Return type: | NoneType |
Returns the search results info for this command invocation.
The search results info object is created from the search results info file associated with the command invocation.
Returns: | Search results info:const:None, if the search results info file associated with the command invocation is inaccessible. |
---|---|
Return type: | SearchResultsInfo or NoneType |
Returns a Splunk service object for this command invocation or None.
The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:
enableheader = true requires_srinfo = true
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
Returns: | splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
---|
Writes a metric that will be added to the search inspector.
Parameters: |
|
---|
The SearchMetric type provides a convenient encapsulation of value. The SearchMetric type provides a convenient encapsulation of value.
Returns: | None. |
---|
Processes search result records and generates a reporting data structure.
Reporting search commands run as either reduce or map/reduce operations. The reduce part runs on a search head and is responsible for processing a single chunk of search results to produce the command’s reporting data structure. The map part is called a streaming preop. It feeds the reduce part with partial results and by default runs on the search head and/or one or more indexers.
You must implement a reduce() method as a generator function that iterates over a set of event records and yields a reporting data structure. You may implement a map() method as a generator function that iterates over a set of event records and yields dict or list(dict) instances.
Configure the map() operation using a Configuration decorator on your map() method. Configure it like you would a StreamingCommand. Configure the reduce() operation using a Configuration decorator on your ReportingCommand() class.
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.
Represents the configuration settings for a ReportingCommand.
Override this method to compute partial results.
Parameters: | records – |
---|
You must override this method, if requires_preop=True.
Process data.
Parameters: |
|
---|---|
Returns: | None |
Return type: | NoneType |
Override this method to produce a reporting data structure.
You must override this method.
Returns the configuration settings for this command.
Returns the fieldnames specified as argument to this command.
Flushes the output buffer and signals that this command has finished processing data.
Returns: | None |
---|
Flushes the output buffer.
Returns: | None |
---|
Returns the input header for this command.
Returns: | The input header for this command. |
---|---|
Return type: | InputHeader |
Returns the logger for this command.
Returns: | The logger for this command. |
---|---|
Return type: |
Syntax: logging_configuration=<path>
Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.
Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]
Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.
Returns the options specified as argument to this command.
Returns the search results info for this command invocation.
The search results info object is created from the search results info file associated with the command invocation.
Returns: | Search results info:const:None, if the search results info file associated with the command invocation is inaccessible. |
---|---|
Return type: | SearchResultsInfo or NoneType |
Returns a Splunk service object for this command invocation or None.
The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:
enableheader = true requires_srinfo = true
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
Returns: | splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
---|
Writes a metric that will be added to the search inspector.
Parameters: |
|
---|
The SearchMetric type provides a convenient encapsulation of value. The SearchMetric type provides a convenient encapsulation of value.
Returns: | None. |
---|
Applies a transformation to search results as they travel through the streams pipeline.
Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of up to 50,000 records. Hence, a search command must be prepared to be invoked many times during the course of pipeline processing. Each invocation should produce a set of results independently usable by downstream processors.
By default Splunk may choose to run a streaming command locally on a search head and/or remotely on one or more indexers concurrently. The size and frequency of the search result batches sent to the command will vary based on scheduling considerations.
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires Splunk 6.3 or later.
Represents the configuration settings that apply to a StreamingCommand.
Process data.
Parameters: |
|
---|---|
Returns: | None |
Return type: | NoneType |
Generator function that processes and yields event records to the Splunk stream pipeline.
You must override this method.
Returns the configuration settings for this command.
Returns the fieldnames specified as argument to this command.
Flushes the output buffer and signals that this command has finished processing data.
Returns: | None |
---|
Flushes the output buffer.
Returns: | None |
---|
Returns the input header for this command.
Returns: | The input header for this command. |
---|---|
Return type: | InputHeader |
Returns the logger for this command.
Returns: | The logger for this command. |
---|---|
Return type: |
Syntax: logging_configuration=<path>
Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.
Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]
Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.
Returns the options specified as argument to this command.
Prepare for execution.
This method should be overridden in search command classes that wish to examine and update their configuration or option settings prior to execution. It is called during the getinfo exchange before command metadata is sent to splunkd.
Returns: | None |
---|---|
Return type: | NoneType |
Returns the search results info for this command invocation.
The search results info object is created from the search results info file associated with the command invocation.
Returns: | Search results info:const:None, if the search results info file associated with the command invocation is inaccessible. |
---|---|
Return type: | SearchResultsInfo or NoneType |
Returns a Splunk service object for this command invocation or None.
The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:
enableheader = true requires_srinfo = true
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
Returns: | splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
---|
Writes a metric that will be added to the search inspector.
Parameters: |
|
---|
The SearchMetric type provides a convenient encapsulation of value. The SearchMetric type provides a convenient encapsulation of value.
Returns: | None. |
---|
Defines the configuration settings for a search command.
Documents, validates, and ensures that only relevant configuration settings are applied. Adds a name class variable to search command classes that don’t have one. The name is derived from the name of the class. By convention command class names end with the word “Command”. To derive name the word “Command” is removed from the end of the class name and then converted to lower case for conformance with the Search command style guide
Represents a search command option.
Required options must be specified on the search command line.
Example:
Short form (recommended). When you are satisfied with built-in or custom validation behaviors.
1 2 3 4 5 6 7 8 | from splunklib.searchcommands.decorators import Option
from splunklib.searchcommands.validators import Fieldname
total = Option(
doc=''' **Syntax:** **total=***<fieldname>*
**Description:** Name of the field that will hold the computed
sum''',
require=True, validate=Fieldname())
|
Example:
Long form. Useful when you wish to manage the option value and its deleter/getter/setter side-effects yourself. You must provide a getter and a setter. If your Option requires destruction you must also provide a deleter. You must be prepared to accept a value of None which indicates that your Option is unset.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | from splunklib.searchcommands import Option
@Option()
def logging_configuration(self):
""" **Syntax:** logging_configuration=<path>
**Description:** Loads an alternative logging configuration file for a command invocation. The logging
configuration file must be in Python ConfigParser-format. The *<path>* name and all path names specified in
configuration are relative to the app root directory.
"""
return self._logging_configuration
@logging_configuration.setter
def logging_configuration(self, value):
if value is not None
logging.configure(value)
self._logging_configuration = value
def __init__(self)
self._logging_configuration = None
|
Validates Boolean option values.
Validates duration option values.
Validates file option values.
Validates integer option values.
Validates regular expression option values.
Validates set option values.