Design Notes
Commands are constrained to this ABNF grammar:
command = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
command-name = alpha *( alpha / digit )
option = option-name [wsp] "=" [wsp] option-value
option-name = alpha *( alpha / digit / "_" )
option-value = word / quoted-string
word = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
quoted-string = dquote *( word / wsp / "" dquote / dquote dquote ) dquote
field-name = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )
It is Constrained to an 8-bit character set. It does not show that field-name values may be comma-separated. This is because Splunk strips commas from the command line. A search command will never see them.
Commands must be statically configured as follows:
1 2 3 4 | [commandname]
filename = commandname.py
supports_getinfo = true
supports_rawargs = true
|
No other static configuration is required or expected and may interfere with command execution.
Commands support dynamic probing for settings.
Splunk probes for settings dynamically when supports_getinfo=true. You must add this line to the commands.conf stanza for each of your search commands.
Commands do not support parsed arguments on the command line.
Splunk parses arguments when supports_rawargs=false. The SearchCommand class sets this value unconditionally. You cannot override it.
Rationale
Splunk parses arguments by stripping quotes, nothing more. This may be useful in some cases, but doesn’t work well with our chosen grammar.
Commands consume input headers.
An input header is provided by Splunk when enableheader=true. The SearchCommand class sets this value unconditionally. You cannot override it.
Commands produce an output messages header.
Splunk expects a command to produce an output messages header when outputheader=true. The SearchCommand class sets this value unconditionally. You cannot override it.
Commands support multi-value fields.
Multi-value fields are provided and consumed by Splunk when supports_multivalue=true. This value is fixed. You cannot override it.
This module represents all fields on the output stream in multi-value format.
Splunk recognizes two kinds of data: value and list(value). The multi-value format represents these data in field pairs. Given field name the multi-value format calls for the creation of this pair of fields.
| Field name | Field data |
|---|---|
| name | Value or text from which a list of values was derived. |
| __mv_name | Empty, if field represents a value; otherwise, an encoded list(value). Values in the list are wrapped in dollar signs ($) and separated by semi-colons (;). Dollar signs ($) within a value are represented by a pair of dollar signs ($$). |
Serializing data in this format enables streaming and reduces a command’s memory footprint at the cost of one extra byte of data per field per record and a small amount of extra processing time by the next command in the pipeline.
A ReportingCommand must override reduce() and may override map(). Map/reduce commands on the Splunk processing pipeline are distinguished as this example illustrates.
Splunk command
sum total=total_date_hour date_hour
Map command line
sum __GETINFO__ __map__ total=total_date_hour date_hour
sum __EXECUTE__ __map__ total=total_date_hour date_hour
Reduce command line
sum __GETINFO__ total=total_date_hour date_hour
sum __EXECUTE__ total=total_date_hour date_hour
The __map__ argument is introduced by ReportingCommand._execute(). Search command authors cannot influence the contents of the command line in this release.
References
Instantiates and executes a search command class
This function implements a conditional script stanza based on the value of module_name:
if module_name is None or module_name == '__main__':
# execute command
Call this function at module scope with module_name=__name__, if you would like your module to act as either a reusable module or a standalone program. Otherwise, if you wish this function to unconditionally instantiate and execute command_class, pass None as the value of module_name.
| Parameters: |
|
|---|---|
| Returns: | None |
Example
1 2 3 4 5 6 7 8 | #!/usr/bin/env python
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class SomeStreamingCommand(StreamingCommand):
...
def stream(records):
...
dispatch(SomeStreamingCommand, module_name=__name__)
|
Dispatches the SomeStreamingCommand, if and only if __name__ is equal to '__main__'.
Example
1 2 3 4 5 6 7 | from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class SomeStreamingCommand(StreamingCommand):
...
def stream(records):
...
dispatch(SomeStreamingCommand)
|
Unconditionally dispatches SomeStreamingCommand.
Generates events based on command arguments.
Generating commands receive no input and must be the first command on a pipeline. By default Splunk will run your command locally on a search head:
@Configuration()
class SomeGeneratingCommand(GeneratingCommand)
You can change the default behavior by configuring your generating command for event streaming:
@Configuration(streaming=True)
class SomeGeneratingCommand(GeneratingCommand)
...
Splunk will then run your command locally on a search head and/or remotely on one or more indexers.
You can tell Splunk to run your streaming-enabled generating command locally on a search head, never remotely on indexers:
@Configuration(local=True, streaming=True)
class SomeGeneratingCommand(GeneratingCommand)
...
If your generating command produces event records in time order, you must tell Splunk to ensure correct behavior:
@Configuration(generates_timeorder=True)
class SomeGeneratingCommand(GeneratingCommand)
...
| Variables: |
|
|---|
Represents the configuration settings for a GeneratingCommand class
Specifies whether output should be used to change the column ordering of fields.
Default: True
Specifies whether required_fields are the only fields required by subsequent commands.
If True, required_fields are the only fields required by subsequent commands. If False, required_fields are additive to any fields that may be required by subsequent commands. In most cases False is appropriate for streaming commands and True is appropriate for reporting commands.
Default: False
Signals that this command expects header information.
Fixed: True
Specifies whether this command generates events in descending time order.
Default: False
Signals that this command generates new events.
Fixed: True
Specifies whether this command should only be run on the search head.
This setting is used to override Splunk’s default policy for running streamable search commands. See the streaming configuration setting.
Default: False
Specifies the maximum number of events that may be passed to an invocation of this command.
This limit may not exceed the value of maxresultrows as defined in limits.conf (default: 50,000). Use a value of zero (0) to select a limit of maxresultrows.
Default: 0
Specifies whether or not this search command must be called with intermediate empty search results.
Default: True
Signals that the output of this command is a messages header followed by a blank line and splunk_csv search results.
Fixed: True
Specifies whether or not this search command requires an authentication token on the start of input.
Default: False
Tells Splunk to issue a performance warning message if more than this many input events are passed to this search command.
A value of zero (0) disables performance warning messages.
Default: 0
Specifies a comma-separated list of required field names.
This list is computed as the union of the set of fieldnames and fieldname-valued options given as argument to this command.
Specifies whether or not this command requires search results information.
If True the full path to a search results information file is provided by SearchCommand.input_header['infoPath'].
Default: False
Specifies whether this command retains _raw events or transforms them.
Default: False
Tells Splunk whether to run this command when generating results for preview rather than final output.
Default: True
Tells Splunk what to do with messages logged to stderr.
Specify one of these string values:
| Value | Meaning |
|---|---|
| 'log' | Write messages to the job’s search.log file |
| 'message' | Write each line of each message as a search info message |
| 'none' | Discard all messages logged to stderr |
Default: 'log'
Specifies that this command is streamable.
By default streamable search commands may be run on the search head or one or more indexers, depending on performance and scheduling considerations. This behavior may be overridden by setting local=True. This forces a streamable command to be run on the search head.
Fixed: True
Signals that this search command supports multivalues.
Fixed: True
Signals that this search command parses raw arguments.
Fixed: True
A generator that yields records to the Splunk processing pipeline
You must override this method.
Processes search results as specified by command arguments.
| Parameters: |
|
|---|
Returns the configuration settings for this command.
Returns the fieldnames specified as argument to this command.
Syntax: logging_configuration=<path>
Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.
Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]
Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.
Returns the options specified as argument to this command.
Returns the search results info for this command invocation or None.
The search results info object is created from the search results info file associated with the command invocation. Splunk does not pass the location of this file by default. You must request it by specifying these configuration settings in commands.conf:
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
| Returns: | SearchResultsInfo, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
|---|
Returns a Splunk service object for this command invocation or None.
The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
| Returns: | splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
|---|
Processes search results and generates a reporting data structure.
Reporting search commands run as either reduce or map/reduce operations. The reduce part runs on a search head and is responsible for processing a single chunk of search results to produce the command’s reporting data structure. The map part is called a streaming preop. It feeds the reduce part with partial results and by default runs on the search head and/or one or more indexers.
You must implement a reduce() method as a generator function that iterates over a set of event records and yields a reporting data structure. You may implement a map() method as a generator function that iterates over a set of event records and yields dict or list(dict) instances.
ReportingCommand configuration
Configure the map() operation using a Configuration decorator on your map() method. Configure it like you would a StreamingCommand.
Configure the reduce() operation using a Configuration decorator on your ReportingCommand() class.
| Variables: |
|
|---|
Represents the configuration settings for a ReportingCommand.
Specifies whether output should be used to change the column ordering of fields.
Default: True
Specifies whether required_fields are the only fields required by subsequent commands.
If True, required_fields are the only fields required by subsequent commands. If False, required_fields are additive to any fields that may be required by subsequent commands. In most cases False is appropriate for streaming commands and True is appropriate for reporting commands.
Default: True
Signals that this command expects header information.
Fixed: True
Signals that this command does not generate new events.
Fixed: False
Specifies the maximum number of events that may be passed to an invocation of this command.
This limit may not exceed the value of maxresultrows as defined in limits.conf (default: 50,000). Use a value of zero (0) to select a limit of maxresultrows.
Default: 0
Specifies whether or not this search command must be called with intermediate empty search results.
Default: True
Signals that the output of this command is a messages header followed by a blank line and splunk_csv search results.
Fixed: True
Specifies whether or not this search command requires an authentication token on the start of input.
Default: False
Tells Splunk to issue a performance warning message if more than this many input events are passed to this search command.
A value of zero (0) disables performance warning messages.
Default: 0
Specifies a comma-separated list of required field names.
This list is computed as the union of the set of fieldnames and fieldname-valued options given as argument to this command.
Indicates whether ReportingCommand.map() is required for proper command execution.
If True, ReportingCommand.map() is guaranteed to be called. If False, Splunk considers it to be an optimization that may be skipped.
Default: False
Specifies whether or not this command requires search results information.
If True the full path to a search results information file is provided by SearchCommand.input_header['infoPath'].
Default: False
Signals that ReportingCommand.reduce() transforms _raw events to produce a reporting data structure.
Fixed: False
Tells Splunk whether to run this command when generating results for preview rather than final output.
Default: True
Tells Splunk what to do with messages logged to stderr.
Specify one of these string values:
| Value | Meaning |
|---|---|
| 'log' | Write messages to the job’s search.log file |
| 'message' | Write each line of each message as a search info message |
| 'none' | Discard all messages logged to stderr |
Default: 'log'
Signals that ReportingCommand.reduce() runs on the search head.
Fixed: False
Denotes the requested streaming preop search string.
Computed.
Signals that this search command supports multivalues.
Fixed: True
Signals that this search command parses raw arguments.
Fixed: True
Override this method to compute partial results.
You must override this method, if requires_preop=True.
Processes search results as specified by command arguments.
| Parameters: |
|
|---|
Override this method to produce a reporting data structure.
You must override this method.
Returns the configuration settings for this command.
Returns the fieldnames specified as argument to this command.
Syntax: logging_configuration=<path>
Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.
Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]
Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.
Returns the options specified as argument to this command.
Returns the search results info for this command invocation or None.
The search results info object is created from the search results info file associated with the command invocation. Splunk does not pass the location of this file by default. You must request it by specifying these configuration settings in commands.conf:
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
| Returns: | SearchResultsInfo, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
|---|
Returns a Splunk service object for this command invocation or None.
The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
| Returns: | splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
|---|
Applies a transformation to search results as they travel through the processing pipeline.
Streaming commands typically filter, sort, modify, or combine search results. Splunk will send search results in batches of up to 50,000 records. Hence, a search command must be prepared to be invoked many times during the course of pipeline processing. Each invocation should produce a set of results independently usable by downstream processors.
By default Splunk may choose to run a streaming command locally on a search head and/or remotely on one or more indexers concurrently. The size and frequency of the search result batches sent to the command will vary based on scheduling considerations. Streaming commands are typically invoked many times during the course of pipeline processing.
You can tell Splunk to run your streaming command locally on a search head, never remotely on indexers.
@Configuration(local=False)
class SomeStreamingCommand(StreamingCommand):
...
If your streaming command modifies the time order of event records you must tell Splunk to ensure correct behavior.
@Configuration(overrides_timeorder=True)
class SomeStreamingCommand(StreamingCommand):
...
| Variables: |
|
|---|
Represents the configuration settings that apply to a StreamingCommand.
Specifies whether output should be used to change the column ordering of fields.
Default: True
Specifies whether required_fields are the only fields required by subsequent commands.
If True, required_fields are the only fields required by subsequent commands. If False, required_fields are additive to any fields that may be required by subsequent commands. In most cases False is appropriate for streaming commands and True is appropriate for reporting commands.
Default: False
Signals that this command expects header information.
Fixed: True
Signals that this command does not generate new events.
Fixed: False
Specifies whether this command should only be run on the search head.
Default: False
Specifies the maximum number of events that may be passed to an invocation of this command.
This limit may not exceed the value of maxresultrows as defined in limits.conf (default: 50,000). Use a value of zero (0) to select a limit of maxresultrows.
Default: 0
Specifies whether or not this search command must be called with intermediate empty search results.
Default: True
Signals that the output of this command is a messages header followed by a blank line and splunk_csv search results.
Fixed: True
Specifies whether this command changes the time ordering of event records.
Default: False
Specifies whether or not this search command requires an authentication token on the start of input.
Default: False
Tells Splunk to issue a performance warning message if more than this many input events are passed to this search command.
A value of zero (0) disables performance warning messages.
Default: 0
Specifies a comma-separated list of required field names.
This list is computed as the union of the set of fieldnames and fieldname-valued options given as argument to this command.
Specifies whether or not this command requires search results information.
If True the full path to a search results information file is provided by SearchCommand.input_header['infoPath'].
Default: False
Specifies whether this command retains _raw events or transforms them.
Default: True
Tells Splunk whether to run this command when generating results for preview rather than final output.
Default: True
Tells Splunk what to do with messages logged to stderr.
Specify one of these string values:
| Value | Meaning |
|---|---|
| 'log' | Write messages to the job’s search.log file |
| 'message' | Write each line of each message as a search info message |
| 'none' | Discard all messages logged to stderr |
Default: 'log'
Signals that this command is streamable.
By default streamable commands may be run on the search head or one or more indexers, depending on performance scheduling considerations. This behavior may be overridden by setting local=True. This forces a streamable command to be run on the search head.
Fixed: True.
Signals that this search command supports multivalues.
Fixed: True
Signals that this search command parses raw arguments.
Fixed: True
Processes search results as specified by command arguments.
| Parameters: |
|
|---|
Generator function that processes and yields event records to the Splunk processing pipeline.
You must override this method.
Returns the configuration settings for this command.
Returns the fieldnames specified as argument to this command.
Syntax: logging_configuration=<path>
Description: Loads an alternative logging configuration file for a command invocation. The logging configuration file must be in Python ConfigParser-format. Path names are relative to the app root directory.
Syntax: logging_level=[CRITICAL|ERROR|WARNING|INFO|DEBUG|NOTSET]
Description: Sets the threshold for the logger of this command invocation. Logging messages less severe than logging_level will be ignored.
Returns the options specified as argument to this command.
Returns the search results info for this command invocation or None.
The search results info object is created from the search results info file associated with the command invocation. Splunk does not pass the location of this file by default. You must request it by specifying these configuration settings in commands.conf:
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
| Returns: | SearchResultsInfo, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
|---|
Returns a Splunk service object for this command invocation or None.
The service object is created from the Splunkd URI and authentication token passed to the command invocation in the search results info file. This data is not passed to a command invocation by default. You must request it by specifying this pair of configuration settings in commands.conf:
The enableheader setting is true by default. Hence, you need not set it. The requires_srinfo setting is false by default. Hence, you must set it.
| Returns: | splunklib.client.Service, if enableheader and requires_srinfo are both true. Otherwise, if either enableheader or requires_srinfo are false, a value of None is returned. |
|---|
Defines the configuration settings for a search command.
Documents, validates, and ensures that only relevant configuration settings are applied. Adds a name class variable to search command classes that don’t have one. The name is derived from the name of the class. By convention command class names end with the word “Command”. To derive name the word “Command” is removed from the end of the class name and then converted to lower case for conformance with the Search command style guide
Represents a search command option.
Required options must be specified on the search command line.
Example:
Short form (recommended). When you are satisfied with built-in or custom validation behaviors.
1 2 3 4 5 | total = Option(
doc=''' **Syntax:** **total=***<fieldname>*
**Description:** Name of the field that will hold the computed
sum''',
require=True, validate=validator.Fieldname())
|
Example:
Long form. Useful when you wish to manage the option value and its deleter/ getter/setter side-effects yourself. You must provide a getter and a setter. If your Option requires destruction you must also provide a deleter. You must be prepared to accept a value of None which indicates that your Option is unset.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | @Option()
def logging_configuration(self):
""" **Syntax:** logging_configuration=<path>
**Description:** Loads an alternative logging configuration file for
a command invocation. The logging configuration file must be in
Python ConfigParser-format. The *<path>* name and all path names
specified in configuration are relative to the app root directory.
"""
return self._logging_configuration
@logging_configuration.setter
def logging_configuration(self, value):
if value is not None
logging.configure(value)
self._logging_configuration = value
def __init__(self)
self._logging_configuration = None
|
Validates Boolean option values.
Validates duration option values.
Validates field name option values.
Validates file option values.
Validates integer option values.
Validates regular expression option values.
Validates set option values.
Base class for validators that check and format search command options.
You must inherit from this class and override Validator.__call__ and Validator.format. Validator.__call__ should convert the value it receives as argument and then return it or raise a ValueError, if the value will not convert.
Validator.format should return a human readable version of the value it receives as argument the same way str does.