Install and configure the Splunk Firehose Nozzle for VMware Tanzu

This topic describes how to install and configure the Splunk Firehose Nozzle for VMware Tanzu.

Prerequisites

For information about Tanzu Operations Manager, see the Using the Tanzu Operations Manager interface topic in the VMware Tanzu Operations Manager documentation. There is no upgrade path from the previous release of Splunk Nozzle for VMware Tanzu (Beta). Install this version to an environment without the Splunk Firehose Nozzle for VMware Tanzu (Beta).

Installation Steps

1. Download the Splunk Firehose Nozzle file from VMware Tanzu Network.
2. Navigate to the Tanzu Operations Manager (Ops Manager) Installation Dashboard and click Import a Product to upload the product file.
3. Click Add next to the uploaded Splunk Firehose Nozzle for VMware Tanzu tile in the Ops Manager Available Products view to add it to your staging area.
4. Click the newly added Splunk Firehose Nozzle for VMware Tanzu tile.
5. Configure the tile as described in the Configuration section of this topic.
6. Click Save.
7. Return to the Ops Manager Installation Dashboard and click Apply Changes to install the Splunk Firehose Nozzle for VMware Tanzu tile.

Configure the Splunk Firehose Nozzle for VMware Tanzu

Perform the following steps to configure the Splunk Firehose Nozzle for VMware Tanzu

1. Select Assign AZs and Networks: Choose placement accordingly.
2. Select Splunk Settings, and then complete the following fields:
   • HTTP Event Collector Endpoint (HEC) URL: HTTP Event Collector endpoint URL.

     <http(s)>://<address>:<port>

     , where address is either FQDN or IP address, and <port> is the port on which Splunk HEC is listening. In a clustered environment this can also be the address and port of a load balancer that distributes requests to multiple configured HEC endpoints. These endpoints can be across both Splunk Heavy Forwarders or Splunk Indexers.
   • HTTP Event Collector (HEC) Token: Splunk generated HEC token. For more information, see the Set up and use HTTP Event Collector in Splunk Web topic in the Splunk platform documentation.
   • Skip SSL validation: Skip SSL certificate validation for connections to your Splunk platform deployment. Secure communications will not check SSL certificates against a trusted Certificate Authority. Do not use for a production environment.
   • Index: The Splunk platform index that events are sent to.

     Setting an invalid index causes events to be lost. This index must match one of the selected indexes for the Splunk HTTP event collector token used for the HTTP Event Collector Token parameter.

3. Select Cloud Foundry Settings, and then complete the following fields:
   • API Endpoint: Cloud Foundry API endpoint.
   • Client ID/ Client Secret: UAA client-id and client-secret. The client needs authorities cloud_controller.admin_read_only and doppler.firehose and grant types client_credentials and refresh_token. Use the uaac tool to create the appropriate client. The uaac commands to add a client are as follows:

       uaac target https://uaa.[system domain url]
       uaac token client get admin -s [admin client credentials secret]
       uaac client add splunk-nozzle --name splunk-nozzle
       uaac client add splunk-nozzle --secret [your_client_secret]
       uaac client add splunk-nozzle --authorized_grant_types client_credentials,refresh_token
       uaac client add splunk-nozzle --authorities doppler.firehose,cloud_controller.admin_read_only
     

4. Skip SSL validation: Skip SSL certificate validation for connections to Cloud Foundry. Secure communications do not check SSL certificates against a trusted Certificate Authority. Do not use for a production environment.
5. Event Types: Choose which events the nozzle forwards to the Splunk indexer(s). To learn more about the following events, see the Protocol Documentation.
   • HttpStartStop
   • LogMessage
   • ValueMetric
   • CounterEvent
   • Error
   • ContainerMetric
6. Select Advanced. Complete the following fields:
   • Scale Out Nozzle: Scale out Splunk Nozzle. Splunk recommends running two or more nozzles for high availability.
   • Firehose Subscription ID: The Loggregator does a round-robin of events across connections having the same ID. An operator would only need to change this if some other connection is also using the default value.
   • Additional Fields: Arbitrary set of key:value pairs in the form key1:value1,key2:value2,key3:value3 that do not occur in event payload itself. These fields are indexed with every event payload, and are used as metadata for indexing and searching. One situation in which this might be useful is differentiating logs from multiple foundations going into the same Splunk Enterprise index.
   • Add App Information: Enriches raw events with app metadata fields - app name, org name, org GUID, space name, space GUID. When configured, the nozzle calls back to the Cloud Foundry API and queries events that have an app GUID to add additional information. This causes a one-time load increase on the API machines proportional the number of running apps, because each Nozzle populates a cache of app metadata.
   • Add Tags: Add additional tags from envelope to splunk event. (Default: false)

     Adding tags / Enabling this feature may slightly impact the performance due to the increased event size).

   • Enable Event Tracing: Enables event trace logging. Splunk events now contain a UUID, Splunk Nozzle Event Counts, and a Subscription-ID for Splunk correlation searches.
   • Monitor Queue Pressure: Time interval (in s/m/h. For example, 3600s or 60m or 1h) for monitoring memory queue pressure. Use to help with back-pressure insights. (Increases CPU load. Use for insights purposes only) Default is 0s (Disabled).
   • HEC Retries: The retry count for sending events to the Splunk platform. Events not successfully sent after this number of retries will be dropped, causing data loss.
   • HEC Batch Size: The number of events per batch sent to Splunk HTTP Event Collector.
   • HEC Workers: The number of concurrent workers sending data to Splunk HTTP Event Collector. Scale this number to your Splunk platform data collection capacity accordingly.
   • Consumer Queue Size: The internal consumer queue buffer size. Events will be sent to your Splunk platform after queue is full.
   • Flush Interval: Time interval (in s/m/h. For example, 3600s or 60m or 1h) for flushing queue to the Splunk platform regardless of Consumer Queue Size. Prevents stale events in low throughput systems.
   • Missing App Cache Invalidate TTL: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the missing app info cache. Set to 0 to maintain cache until nozzle restart.
   • App Cache Invalidate TTL: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the app info local cache. Set to 0 to only populate the cache during startup or restart of the nozzle.
   • Org Space Cache Invalidate TTL: Time interval (in s/m/h. For example, 3600s or 60m or 1h) between refreshing the org and space cache.
   • Ignore Missing App: If the application is missing, then stop repeatedly querying application info from Cloud Foundry.
   • App Limits: The number of apps for which metadata is gathered when refreshing the app metadata cache (order based on app creation date). Set to 0 to remove limit.
   • Nozzle Memory: Configure memory for Splunk Firehose Nozzle application in MB.
   • Firehose Keep Alive: Keep alive duration (in s/m/h. For example, 3600s or 60m or 1h) for the firehose consumer.
   • Drop Warn Threshold: Threshold for the count of dropped events in case the downstream is slow. Based on the threshold, the errors will be logged.
   • Resource Config: Select default options.
   • Stemcell: Ensure the proper stemcell is available.