Performance and Sizing

Splunk Inc. has tested the Splunk Firehose Nozzle for VMware Tanzu Architecture running as a single deployed nozzle on the AWS instance type c4.4xlarge. This EC2 instance type has 8 CPU and 32 GB memory. Storage is EBS-Only and has dedicated EBS Bandwidth of 2,000 Mbps. Tests were performed with structured and unstructured data with 2 different event sizes 256 and 1024 bytes. Averaged performance metrics over an extended period of time resulted in the following results:

Performance results should be taken as a guide only as different configurations and environments may vary results.

Tests	Events per Second (EPS)	Mbps	Nozzle CPU (%)	Nozzle Mem Total (MB)
Structured Data (JSON) - 256 byte event size	9444	7.30	530.6	57.2
Structured Data (JSON) - 1024 byte event size	6775	10.90	567.9	79.9
Unstructured Data - 256 byte event size	11189	9.10	502.3	58.3
Unstructured Data - 1024 byte event size	9487	16.03	491.5	82.8

Nozzle CPU (%) 100% means that a single core is wholly dedicated to the Nozzle. 500% means five cores are wholly dedicated. Factoring these optimal testing numbers, Splunk recommends that to run a single Splunk nozzle instance, you use a system with at least 8 CPUs and 300 MB RAM.

Index routing

Use index routing to send different Cloud Foundry logs to different indexes for better Audit Command Language (ACL) and data retention control in your Splunk platform deployment. Configure application index routing via application manifest Navigate to your deployment's your app manifest, and provide an environment variable named SPLUNK_INDEX and assign it the index you would like to send the app data to. Make sure Splunk nozzle is configured with Add App Info (Select at least one of AppName,OrgName,OrgGuid,SpaceName,SpaceGuid) to enable app info caching.

applications:
 - name: console
  memory: 256M
  disk_quota: 256M
  host: console
  timeout: 180
  buildpack: go
  health-check-type: port
  env:
    SPLUNK_INDEX: testing_index

 

Verify SPLUNK_INDEX exists on your Splunk platform deployment. Otherwise data will be lost.

Index routing via Splunk configuration

Logs can be routed using fields such as app ID/name, space ID/name or org ID/name. Configure the Splunk configuration files props.conf and transforms.conf on your Splunk platform indexers or Splunk platform heavy forwarders, if deployed. The following are examples of index routing configurations:
Example: Route data from application ID 95930b4e-c16c-478e-8ded-5c6e9c5981f8 to a Splunk production index:

$SPLUNK_HOME/etc/system/local/props.conf [cf:logmessage]   TRANSFORMS-index_routing = route_data_to_index_by_field_cf_app_id

$SPLUNK_HOME/etc/system/local/transforms.conf [route_data_to_index_by_field_cf_app_id]   REGEX = "(\w+)":"95930b4e-c16c-478e-8ded-5c6e9c5981f8"   DEST_KEY = _MetaData:Index   FORMAT = prod

Example: Route application logs from any Cloud Foundry organizations whose names are prefixed with sales to a Splunk software sales index.

$SPLUNK_HOME/etc/system/local/props.conf ``` [cf:logmessage] TRANSFORMS-index_routing = route_data_to_index_by_field_cf_org_name

*$SPLUNK_HOME/etc/system/local/transforms.conf*
[route_data_to_index_by_field_cf_org_name] REGEX = "cf_org_name":"(sales.*)" DEST_KEY = _MetaData:Index FORMAT = sales ```

Example: Route data from sourcetype cf:splunknozzle to index new_index:

$SPLUNK_HOME/etc/system/local/props.conf [cf:splunknozzle]   TRANSFORMS-route_to_new_index = route_to_new_index

$SPLUNK_HOME/etc/system/local/transforms.conf [route_to_new_index]   SOURCE_KEY = MetaData:Sourcetype   DEST_KEY =_MetaData:Index   REGEX = (sourcetype::cf:splunknozzle)   FORMAT = new_index