Splunk® Data Stream Processor

Function Reference

Acrobat logo Download manual as PDF

Acrobat logo Download topic as PDF

Functions by category

Use the following section as a reference of available streaming and scalar functions.

Streaming functions

These functions are visible in the UI. Streaming functions are full nodes in the pipeline with edges that data flows through. Data streams from one node (the streaming function) through its edge to the next node (streaming function). Streaming functions can use scalar functions.

Functions Description
Adaptive Thresholding Detects anomalies on the stream and dynamically generates threshold values based on observed data values.
Aggregate and Trigger Triggers an event output based on a custom condition over a set of aggregated events.
Apply Line Break Break and merge events sent from a universal forwarder.
Apply ML Model Use to see how a trained model performs in the DSP environment.
Apply Timestamp Extraction Extract a timestamp using a designated extraction type.
Batch Records Batches records before sending them to an index or a third-party sink.
Bin Puts continuous numerical values into discrete sets, or bins, by adjusting the value of <field> so that all of the items in a particular set have the same value.
Break Events Break events using a Java regular expression as a delimiter.
Datagen Function under ongoing development. Generates synthetic data.
Drift Detection Identifies large scale shifts and abrupt changes in a time-series data stream.
Eval Calculate an expression and put the resulting value into the record as a new field.
Fields Select a subset of fields from a record.
From Used to retrieve data from a specific source function in the SPL2 Pipeline Builder.
Into Used to send data to a specific sink function in the SPL2 Pipeline Builder.
Key_by Group a stream of records by one or more field(s) and returns a grouped stream.
Lookup Invokes field value lookups.
Merge Events Parses data received from a universal forwarder into a stream of complete events for a Splunk Index.
Mvexpand Expand the values in a multivalue field into separate events, one event for each value in the multivalue field.
Pairwise Categorical Outlier Detection Detects anomalous combinations of values from two categorical variables.
Parse regex (rex) Extract or rename fields using Java regular expression-named capturing groups.
Rename Rename one or more fields.
Select Assigns an alternative name to a field or applies scalar functions to a group of fields.
Sentiment Analysis Classifies raw text as positive, negative, or neutral.
Sequential Outlier Detection Identify anomalous events in time-series sequence data.
Stats Applies one or more aggregation functions on a stream of events in a specified time window.
Time Series Decomposition (STL) Automatically decomposes time series data streams into trend, seasonal, and remainder components in real time.
To Splunk JSON Format records to adhere to the Splunk HEC event JSON or the Splunk HEC metric JSON format.
Union Combines streams with the same input schema into one stream with all of the events of the input streams.
Where Keep records that pass a Boolean function.

Streaming source functions

These functions are visible in the UI and are source functions.

Functions Description
Splunk DSP Firehose Get data sent from Splunk DSP Ingest, Forwarders, and Collect API services.
Forwarders Service Get data from the Splunk Forwarders service.
Ingest Service Get data from the Ingest service.
Amazon CloudWatch Get data from Amazon CloudWatch.
Amazon Kinesis Data Stream Get data from Amazon Kinesis Data Stream.
Amazon Metadata Get data from the resources and infrastructure in Amazon Web Services (AWS).
Amazon S3 Get data from Amazon S3.
Apache Pulsar Get data from Apache Pulsar.
Kafka

Get data from Apache or Confluent Kafka.

Microsoft Azure Event Hubs Get data from Microsoft Azure Event Hubs.
Microsoft Azure Monitor Get data from Microsoft Azure Monitor.
Google Cloud Monitoring Get data from Google Cloud Monitoring.
Google Cloud Pub/Sub Get data from Google Cloud Pub/Sub topics.
Microsoft 365 Get data from the Office 365 Management Activity API.

Streaming sink functions

These functions are visible in the UI and are sink functions.

Sink functions do not have preview data enabled. To see if your index or third-party service is successfully receiving your data, you must search for your data in your index or third-party service.

Functions Description
Send to a Splunk Index with Batching Sends data to an external Splunk Enterprise environment. The Splunk Enterprise Indexes function combines the actions of three underlying DSP functions into one for user convenience: To Splunk JSON, Batch Bytes, Splunk Enterprise.
Send to a Splunk Index Sends data to an external Splunk Enterprise environment.
Send to a Splunk Index (Default for Environment) Sends data to your default, pre-configured Splunk Enterprise instance.
Send to Amazon Kinesis Data Streams Sends data to an Amazon Kinesis Data Stream using an AWS access key and secret key authentication.
Send to Amazon S3 Sends data to Amazon S3.
Send to Kafka Sends data to an Apache Kafka topic using a Kafka connection.
Send to Microsoft Azure Event Hubs (Beta) Sends data to Microsoft Azure Event Hubs.
Send to Null Sends data to a dev/null sink.
Send Metrics Data to SignalFx Sends metric data to SignalFx.
Send Trace Data to SignalFx Sends trace data to SignalFx.

Scalar functions

Scalar functions are not full nodes in a pipeline, but instead, help perform certain operations inside of a streaming function. You can use scalar functions to do things like addition and subtraction, comparisons, conversions, or other similar tasks.

Scalar functions operate in the context of the streaming function that they are called in.

Function Category Function list Description
Casting functions cast Converts an expression from one data type to another.
ucast Provides a way to cast maps and collections, regardless of the data type that the map or collection may contain.
Aggregate functions average Returns the average of the values in the specified field.
count Returns the number of non-null values in a time window.
estdc Calculates an approximated distinct count value for any field.
max Returns the maximum value in a time window.
mean Calculates the average (mean) of values in a time window.
min Returns the minimum value in a time window.
perc Computes the approximate q-th percentile value of a numeric field input field.
sum Returns the sum of values in a time window.
Conditional scalar functions cidrmatch Returns TRUE or FALSE based on whether an IPv4 address matches an IPv4 CIDR notation.
coalesce Takes a variable number of arguments and returns the first value that is not NULL.
if Assigns an expression if the value is true, and another expression if the value is false.
in Returns TRUE if one of the values in a list matches a value in the field you specify.
like Returns TRUE if TEXT matches PATTERN.
null if equal (nullif) Compares two fields and returns NULL if two fields are equal.
validate Returns the first string value corresponding to the first test that evaluates to FALSE.
Conversion scalar functions base64_decode Converts a Base64-encoded string to bytes.
base64_encode Converts a byte array value to a Base64-encoded string.
deserialize_json_object Converts a JSON byte string into a map.
from_json_array Converts a JSON string into an array of the JSON structure, including nested keys.
from json object Converts a JSON string into a map of the JSON structure, including nested keys.
gunzip Decompresses a GZipped byte array.
gzip Returns Gzipped-compressed bytes.
inet_aton Converts a string IPv4 or IPv6 IP address and returns the address as type Long.
inet_ntoa Converts a decimal IP address to dotted-decimal form.
parse_bool Parses a string as a boolean.
parse_double Parses a string and returns the numeric value as a Double.
parse_float Parses a string and returns the numeric value as a Float.
parse_int Parses a string as an integer.
parse long Parses a string and returns the numeric value as Long.
parse_millis Converts Splunk Enterprise "time" format to DSP "timestamp" format.
parse_nanos Converts Splunk Enterprise "time" format to DSP "nanos" format.
serialize_json Converts the current record into a JSON byte string.
serialize_json_collection Converts a map of JSON structure into a JSON byte array.
to_bytes Converts a string to a byte string.
to_json Converts a map of a JSON object's structure to a JSON string.
to_string Converts a byte array to a String.
tostring Converts a number to a string.
Cryptographic scalar functions md5 Computes and returns the MD5 hash of a byte value X.
sha1 Computes and returns the secure hash of a byte value X based on the FIPS compliant SHA-1 hash function.
sha256 Computes and returns the secure hash of a byte value X based on the FIPS compliant SHA-256 hash function.
sha512 Computes and returns the secure hash of a byte value X based on the FIPS compliant SHA-512 hash function.
Date and Time scalar functions relative_time Applies a relative time specifier to a UNIX time value.
strftime This function formats a UNIX timestamp into a human-readable timestamp.
strptime This function parses a date string into a UNIX timestamp.
time This function returns the wall-clock time, in the UNIX time format, with millisecond resolution.
Iterator scalar functions filter Filters elements of a list.
for_each For each element of a list, evaluate an expression Y and return a new list containing the results.
iterator Iterates through a list and temporarily assigns each element in list for use in the iterator scalar functions.
List scalar functions length Returns the character length of a given input.
mvdedup Removes duplicates from a list.
mvappend Takes an arbitrary list of arguments, where each argument is a single string or a list of strings, and returns all elements as a single flattened list.
mvindex Returns the element at the list at the index.
mvjoin Takes all of the values in a list and appends them together delimited by STR.
mvrange Returns a list for a range of numbers.
mvsort Takes a list and returns the list with the elements of the list sorted lexicographically.
split Splits a string using a delimiter.
Map scalar functions contains_key Checks a map for a specified key.
create_map Creates a new map object at runtime.
flatten Flattens a list or a map
length Returns the character length of a given input.
map_delete Removes a key from a map.
map_get Returns the value corresponding to a key in the map input.
map_keys Returns a list of keys in a map.
map_set Insert or overwrite key-value pairs in a map.
map_values Returns all values in a map.
spath Extract a value from a map or collection.
Mathematical scalar functions abs Takes a number and returns its absolute value.
ceil Rounds a number up to the next highest integer.
exp Takes a number value and returns the exponential evalue.
floor Rounds a number down to the nearest whole integer.
log Takes one or two numbers and returns the logarithm of the first argument value using the second argument base as the base.
natural logarithm (ln) Takes a number and returns its natural logarithm.
pi Returns the constant pi to 11 digits of precision.
power of base (pow) Takes two numbers base and exp, and returns baseexp.
random integer (randomint) Returns a random integer in the range of 0 to 231-1.
round value (round) Takes a number value and returns value rounded to the nearest whole number.
round value (round) Takes two numbers, value and num_decimals, and returns value rounded to the amount of decimal places specified by num_decimals.
sqrt Takes a number value and returns its square root.
String manipulation scalar functions concat Combines the first and second strings together.
extract_grok Extracts matching groups with a Grok-compatible pattern.
extract_key_value Extract key-value pairs.
extract_regex Uses a Java regular expression to extract capturing groups from the input.
join Joins a list of strings using a delimiter and returns a single string.
len Returns the character length of a string.
lower Converts a string to lowercase.
ltrim Trims extra characters from the left side.
match_regex Matches inputs against a pattern defined with a Java regular expression.
match_wildcard Matches inputs against a wildcard pattern.
replace Performs a regex replacement on a string.
rtrim Trims extra characters from the right side.
substr Returns a substring of a string.
trim Trim extra characters.
upper Converts a string to uppercase.
url_decode Takes a URL string and returns the unescaped or decoded URL string.
url_encode Encodes a string for the query string parameters in a URL.
Last modified on 28 October, 2020
  NEXT
How to use the Function Reference

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters