Batch Records to optimize throughput to Splunk Enterprise indexes

Multiple records can be transmitted in one request with batching. Batching records, as opposed to packaging each event individually, can increase throughput by reducing the net amount of data being transmitted. To batch records, use the Batch Records streaming function after doing any desired data transformations in your pipeline.

Routing batched data to different indexes

Right now, you can only designate an index to send your data to using the Splunk Enterprise sink function. Because batching applies any function arguments across the entire batched payload, the index you provide in your Write to Splunk function(s) gets assigned to the entire payload. Therefore, if you want to send batched data to specific indexes, you must make one branch per index that you want to send data to.

Example

Prerequisites

• A data pipeline with a source function.
• A properly configured HEC-token with the indexes that you want to send data to assigned.
• A Splunk Enterprise connection, if using the Write Splunk Enterprise function.

Steps

1. After doing any transformations on your data, use the Filter streaming function to filter your records based on what index you want to send your records to.
   
2. Branch your pipeline by clicking the + icon on the function node and add another filter function.
   For example, if you want to send records with sourcetype cisco:asa to one index and records with sourcetype syslog to a different index, use a filter function with eq(get("sourcetype"),"cisco:asa"); and another with eq(get("sourcetype"), "syslog");.
3. Add the Batch Records function to both of your branches.

   Field	Description	Example
   Num records	The maximum number of records to send per batch. Capped at 1000 records.	99;
   Millis	The interval, in milliseconds, at which to send batched records.	1000;

4. End both of your branches with the Write to Index or Write to Splunk Enterprise sink functions.
   a. If you are using the Write to Index function:

   Field	Example
   module	"";
   dataset	"index_name";

   
   b. If you are using the Write to Splunk Enterprise function, select a connection from the drop-down menu and fill out the following fields:

   Field	Example
   index	"index_name";
   Parameters	async = true

   If your index is set in your data's attributes, you can type cast(map-get(get("attributes"), "index"), "string"); to send your data to the index specified in your data.

5. Save and activate your pipeline.

On completion, your pipeline should look like the following.