New features for DSP

Here's what's new in each version of the Splunk Data Stream Processor

Version 1.2.0

What's new in the docs

The DSP documentation was refactored in version 1.2.0 to present information in a more intuitive manner and better reflect the end-to-end user experience of the product. As a result, the titles and locations of some topics have changed.

• The Getting Data In manual has been replaced by the Connecting to Data Sources and Destinations manual, which provides complete information about how to connect your DSP pipeline to a given data source or data destination.
• The contents of the Use the Data Stream Processor manual have been reorganized.
• The contents of the following chapters from the Use the Data Stream Processor manual have been moved into the new Connecting to Data Sources and Destinations manual:
  • "Pipeline requirements for specific data sources in DSP"
  • "Format data in DSP to send to the Splunk platform"
  • "Send data from DSP to other destinations"
• The Source functions (Data Sources) and Sink functions (Data Destinations) topics in the Function Reference manual have been moved and rewritten. The source and sink functions are now in dedicated chapters of the Function Reference manual.
• The Send data from Splunk DSP to SignalFx topic has been rewritten and now includes a detailed example demonstrating how to send metrics.log data from the Splunk universal forwarder to SignalFx.
• Added TLS/Cipher suite information.
• Updated the DSP HEC examples, and added documentation about multi-metric support.

What's new in the Data Stream Processor

The following table describes new features or enhancements in DSP 1.2.0.

New Feature or Enhancement	Description
Support for CSV and Splunk Enterprise KV Store lookups	DSP now supports lookups to Splunk Enterprise KV Stores or CSV files for increased data enrichment. See About lookups and Lookup for more information.
Support for sending data to a Splunk Enterprise KV Store collection	DSP now supports writing data from DSP into a Splunk Enterprise KV Store collection. See Connect Splunk DSP to a Splunk Enterprise KV Store and Write Thru KV Store for more information.
Streaming ML	Streaming ML is the Splunk Enterprise machine learning framework designed specifically for online learning. This framework includes a library of operators that enable users to apply machine learning models to streaming data, without requiring offline batch training jobs. Steaming ML in DSP 1.2 includes three new functions for Time Series Decomposition (STL), Pairwise Categorical Outlier Detection, Percentiles, and more. You must install the Streaming ML plugin to access these functions. All Streaming ML functions are currently in beta. For more information, see About the Streaming ML Plugin.
Apply Line Break	You can now perform line breaking and merging for universal forwarder data in one function. In addition, you can now migrate and reuse existing props.conf line_breaking configurations in DSP. See Apply Line Break for more information.
Apply Timestamp Extraction	You can now extract additional timestamp formats using strptime() and regular expressions. In addition, you can now migrate and reuse existing props.conf timestamp extraction configurations in DSP. See Apply Timestamp Extraction for more information.
Apache Pulsar Connector	DSP now supports collecting data from an Apache Pulsar topic. See Connecting Apache Pulsar to your DSP pipeline as a data source and Get data from Apache Pulsar for more information.
Google Cloud Pub/Sub Connector	DSP now supports collecting messages from Google Cloud Pub/Sub. See Connecting Google Cloud Pub/Sub to your DSP pipeline and Get data from Google Cloud Pub/Sub for more information.
Send to SignalFx (trace)	You can now send trace data to a SignalFx endpoint using the SignalFx connector. See Connecting SignalFx to your DSP pipeline and Send data to SignalFx (trace) for more information.
Updates to the Splunk App for DSP	The DSP Health application has been renamed to Splunk App for DSP. You can now collect additional metrics about your DSP environment and monitor those metrics in Splunk Enterprise. In addition, there are now more dashboards to help you visualize the health of your DSP environment. See About the Splunk App for DSP for more information.
New install flavors and profiles.	DSP now supports additional install flavors and node roles. In addition, DSP also supports more than five master nodes in a cluster.
Updated Send to Microsoft Azure Event Hubs sink function	This sink function now provides improved performance and data batching controls. See Send data to Microsoft Azure Event Hubs (Beta) for more information.
Updated Send to Amazon S3 sink function	You can now compress the data that you send to Amazon S3. When sending data in Parquet format, you can now specify the version of Parquet Writer to use, the maximum size of each row group, and how DSP handles records with invalid schemas. Files generated by this function are now given the correct filename extension based on the file format. See Send data to Amazon S3 for more information.
SPL2 Named Arguments	DSP now supports named arguments when using SPL2 (Search Processing Language version 2) for source, sink, and scalar functions. See the SPL2 examples in the Function Reference manual for more information.
Dot and bracket notation support for accessing lists and maps	It's now easier to access list and maps. See Accessing list elements using bracket notation and Accessing map elements using dot notation.
map_merge scalar function	You can now merge two or more maps together in DSP using the map_merge scalar function. See map_merge for more information.
Improved performance of the Forwarders Service	Changes to the Forwarders Service for better performance.
Updated names for connectors and functions	The display names that appear in the DSP UI for connectors, source functions, and sink functions have been updated for clarity and consistency. Additionally, the SPL2 names for some functions have been updated. See the "Renamed functions in version 1.2.0" section on this page for more information.
SCloud 4	SCloud 4.0 is now bundled with DSP.
--location install flag	You can now specify a location for Gravity to save container and state information using a --location flag.

What's new in the DSP SDK

The following table describes new features or enhancements in the DSP SDK.

New Feature or Enhancement	Description
RuntimeContext#getArgument() no longer replaces dashes in argument names with underscores	Previously, scalar function arguments could be accessed from RuntimeContext using dash-cased argument names. Now, all argument names must be accessed using their underscore_cased names.
Record#get() returns read-only view of maps and lists	Previously, functions were able to read maps or lists from Record and directly modify them. Now, maps or lists read from Record must be explicitly copied before they can be modified.
AggregationFunction#initialState() is deprecated	Update classes that implement AggregationFunction to use AggregationFunction#initialState(RuntimeContext) instead.

Renamed SPL2 functions in version 1.2.0

The following functions were renamed in 1.2.0.

Original SPL2 function name	Updated SPL2 function name
read_event_hubs	event_hubs
read_kafka	kafka
read_kinesis	kinesis
read_splunk_firehose	splunk_firehose
receive_from_forwarders	forwarders
receive_from_ingest_rest_api	ingest_rest_api
write_index	index
write_kafka	kafka
write_kinesis	kinesis
write_null	dev_null

Version 1.1.0

New Feature or Enhancement	Description	Learn more link
SPL2 Support	DSP now supports creating and configuring DSP pipelines using SPL2 (Search Processing Language version 2).	SPL2 for DSP.
SPL2 Builder	DSP now supports an additional pipeline builder experience allowing you to write pipelines in SPL2.	SPL2 Pipeline Builder.
DSP HTTP Event Collector	You can send events and metrics to a DSP data pipeline using the DSP HTTP Event Collector (DSP HEC). The DSP HEC supports the Splunk HTTP Event Collector (HEC) /services/collector, /services/collector/event, and /services/collector/event/1.0 endpoints allowing you to quickly redirect your existing Splunk HEC workflow into DSP via the DSP Firehose.	Send events to a DSP data pipeline using the DSP HTTP Event Collector.
Syslog support	You can now easily ingest syslog data into DSP using Splunk Connect for Syslog (SC4S).	Send Syslog events to a DSP data pipeline using SC4S with DSP HEC.
Amazon Linux 2 support	DSP now supports Amazon Linux 2.	Hardware and Software requirements.
Upgraded Streams REST API	Upgraded Streams REST API endpoints to v3beta1	Splunk Data Stream Processor REST API Reference.
Apache Pulsar messaging bus	DSP now uses Apache Pulsar as its messaging bus for data sent via the Ingest, Collect, and Forwarders Services.	Increase Pulsar partitions for improved pipeline throughput
Splunk Enterprise sink function with Batching	You can now do index-based routing even while batching records. This function performs the common workflow of mapping the DSP event schema to Splunk HEC metrics or events schema, turning records into JSON payloads, and batching the bytes of those payloads for better throughput.	Write to the Splunk platform with Batching
Splunk Enterprise sink function	This function replaces Write Splunk Enterprise. This function adds out of the box support for index-based routing while batching.	Write to the Splunk platform
Batch Bytes streaming function	DSP now supports batching your data as byte payloads for increased throughput.	Batch Bytes
To Splunk JSON streaming function	You can now perform automatic mapping of DSP events schema to Splunk HEC events or metrics schema.	To Splunk JSON.
Write to S3-compatible storage sink function	DSP now supports sending data to an Amazon S3 bucket.	Write to S3-compatible storage
Write to SignalFx sink function	DSP now supports sending data to a SignalFx Endpoint.	Write to SignalFx
Microsoft 365 Connector	DSP now supports collecting data from Microsoft 365 and Office 365 services using the Microsoft 365 Connector.	Use the Microsoft 365 Connector with Splunk DSP.
Google Cloud Monitoring Metrics Connector	DSP now supports collecting metrics data from Google Cloud Monitoring.	Use the Google Cloud Monitoring Metrics Connector with Splunk DSP.
Amazon S3 Connector	The Amazon S3 Connector now supports Parquet format as a File Type.	Use the Amazon S3 Connector with Splunk DSP.
Write to Azure Event Hubs Using SAS Key sink function (Beta)	DSP now supports sending data to an Azure Event Hubs namespace using an SAS key. This is a beta function and not ready for production.	Write to Azure Event Hubs.
Bug fixes	The Splunk Data Stream Processor 1.1.0 includes several bug fixes.	Fixed Issues for DSP.

Version 1.0.1

• Bug fixes. For details, see Fixed issues.

Version 1.0.0

This is the first release of the Splunk Data Stream Processor.