Splunk® Data Stream Processor

Function Reference

Acrobat logo Download manual as PDF

Acrobat logo Download topic as PDF

Aggregate with Trigger

This topic describes how to use the function in the Splunk Data Stream Processor.

Description

Applies one or more aggregation functions on a stream of records in a specified time window, and outputs a record when a pre-defined condition is met. Use this function to set an alert on your data before your data gets indexed. This function accepts a variable number of aggregation functions.

Function Input/Output Schema

Function Input
collection<record<R>>
This function takes in collections of records with schema R.
Function Output
collection<record<S>>
This function outputs the same collection of records but with a different schema S.

Syntax

The required syntax is in bold.

aggregate_and_trigger
aggregations(field) [AS <newfield>]
keys=<field> ["," <field>] [AS <newfield>]
timestamp=<field>
size=<long>
slide=<long>
grace_period=<long>
trigger_count=<long>
trigger_interval=<long>
trigger_time_type=<trigger_type_options>
predicate=<boolean-expression>
trigger_max_fire_count=<long>
custom_fields=<expression> AS <newfield>

Required arguments

Aggregations
Syntax: aggregations=collection<expression<any>>
Description: An aggregation function to apply on your events.
keys
Syntax: <field>
Description: The field values by which to group events.
UI Example: body
Timestamp
Syntax: <field>
Description: The field name where your record's timestamps are located.
UI Example: timestamp or get("timestamp");
Size
Syntax: <long>
Description: The window length in milliseconds to group events.
Example: 10000
Slide
Syntax: <long>
Description: The rolling window time offset.
Example: 360000
Grace Period
Syntax: <long>
Description: The amount of time, in milliseconds, to wait for late-arriving events.
Example: 10000.
Trigger Count
Syntax: <long>
Description: Trigger an event based on the count of aggregated events. For example, if set to 1, then a trigger is fired after every one event. Set to zero to only trigger events at the close of the window. The trigger count is reset after each window.
Example: 1
Trigger Interval
Syntax: <long>
Description: Trigger an event after a certain amount of time has passed since the start of the window. For example, setting this to 3600000 will cause a trigger to fire after 3600000 milliseconds (1 hour) has elapsed since the start of the window. The behavior of this interval depends on the trigger time type. Set to zero to only trigger events at the close of the window.
Example: 0
Trigger Time Type
Syntax: EventTime or ProcessingTime
Description: Determines the time when an event is triggered. Select EventTime to use the timestamp field as the measurement of time. Select ProcessingTime to use the system clock of the machine running the job. See the Trigger Time Type Options section for an example.
Predicate
Syntax: <boolean-expression>
Description: A boolean scalar function that evaluates events and triggers an alert if the condition is true per aggregated event.
Example: total_time_taken>1000L
Trigger Max Fire Count
Syntax: <long>
Description: The maximum number of times that the predicate evaluates to true per aggregated record. Once met, no more records will be outputted for that window.
Example: 1
Custom Fields
Syntax: collection<expression<any>>
Description: A function that runs when a triggered record passes the predicate.
Example: custom_fields=["triggered because status_code = 500 has total_time_taken > 1000" AS message]

If both trigger count and trigger interval are set to positive numbers, whichever occurs first will cause the subsequent triggered event to happen.

Trigger Time Type Options

For example, say if you have two events: one with a timestamp of 1PM and another event with a timestamp of 2PM and your trigger interval is 1 hour. In this example, due to network latency, both events are received in close succession at 2:01PM. In EventTime, an event is triggered because the difference in the event timestamp is greater than the trigger interval of one hour. In ProcessingTime, because the system processes both events within the given trigger interval, no event is triggered.

Alternatively, say if you have two events: one with a timestamp of 1PM and one with a timestamp of 1:01PM. Due to network latency, the second event is received at 2PM. In EventTime, an event is not triggered because the difference in the event timestamp is not greater than the trigger interval. In ProcessingTime, the event is triggered, because the system took longer than the trigger interval to process both events.

EventTime uses the same time unit as the timestamp field. ProcessingTime uses milliseconds.

Usage

The aggregation function has no concept of wall clock time, and the passage of time is based on the timestamps of incoming records. The aggregation function tracks the latest timestamp it received in the stream as the "current" time, and it determines the start and end of windows using this timestamp. Once the difference between the current timestamp and the start timestamp of the current window is greater than the window length, that window is closed and a new window starts. However, since events may arrive out of order, the grace period argument allows the previous window W to remain "open" for a certain period G after its closing timestamp T. Until we receive a record with a timestamp C where C > T + G, any incoming events with timestamp less than T are counted towards the previous window W.

SPL2 example

Count the number of errors within a 1 hour time window and output an alert as soon as five errors are seen.

...| aggregate_and_trigger keys=[host] timestamp=timestamp size=3600000 slide=3600000 grace_period=0 trigger_count=10 trigger_interval=0 trigger_time_type="EventTime" aggregations=[count(cast(map_get(attributes, "error"), "integer")) AS error_code] predicate=error_code>5 trigger_max_fire_count=1 custom_fields=["triggered because error-count > 5" AS action] |...
Last modified on 25 September, 2020
PREVIOUS
Adaptive Thresholding
  NEXT
Apply Line Break

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.1.0, 1.2.0


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters