Configure Cloud Pub/Sub Based Bucket inputs for the Splunk Add-on for Google Cloud Platform

Complete the following steps to configure Cloud Pub/Sub Based Bucket inputs for the Splunk Add-on for Google Cloud Platform using Splunk Web or a configuration file.

1. You must manage Google Credentials for the add-on as a prerequisite.
2. Configure Google services for the Pub/Sub Based Bucket input.
3. Configure Google services permissions for the Pub/Sub Based Bucket input.
4. Configure Cloud Pub/Sub Based Bucket inputs either through Splunk Web or configuration files.

Configuration prerequisites

Before you configure Pub/Sub Based Bucket inputs, perform the following tasks:

• Create a Pub/Sub Topic and Subscription to receive notifications, and a second Pub/Sub Topic and Subscription to serve as a dead letter topic.
• Configure Pub/Sub notifications for the Cloud Storage Bucket to send notifications to the topic on the object creation and updation. This lets the bucket notify the add-on that new events were written to the bucket.
• Add Google Credentials to the Splunk Add-on for Google Cloud Platform.

Attach a Notification Configuration to Bucket to receive notifications

Attach notification configuration to the bucket using glcoud CLI in order to allow your Bucket to send notifications to a Pub/Sub Topic. Please follow gcloud storage buckets notifications create Google documentation for the notification configuration.

Best practices

Please take into consideration the following points while configuring your inputs:

• Pub/Sub Based Bucket input only collects data for objects that meet the following criteria:
  • The input must be stored in the Storage Bucket
  • The input must have a notification sent to Pub/Sub Topic
• To achieve high throughput data ingestion from a Storage Bucket, configure multiple Pub/Sub Based Bucket inputs to scale out data collection.
• Set up a Pub/Sub Dead Letter Topic for the Pub/Sub Topic to be used for the input for storing invalid messages. For information about Pub/Sub Dead Letter Topic and how to configure it, see https://cloud.google.com/pubsub/docs/handling-failures in the Google Cloud Pub/Sub documentation.
• Configure the Subscription Acknowledgement deadline to prevent multiple inputs from receiving and processing messages in a subscription more than once. Though the add-on will modify the Acknowledgement deadline when it is close to expiring for a message in processing, It is recommended to set the Acknowledgement deadline to 5 minutes or longer. If the Acknowledgement deadline for a message is reached before the message is fully processed by the input, the message reappears in the subscription and is retrieved and processed again, resulting in duplicate data.

Configure Cloud Storage Bucket inputs using the Splunk Web

Follow these steps to configure Cloud Pub/Sub Based Bucket inputs.

1. Follow the menu path Create New Input > Cloud Storage Bucket > Cloud Pub/Sub Based Bucket.
2. Enter the Name, Credentials, Projects, Pub/Sub Subscription, Index, Sourcetype, Message Batch Size, and Number of Threads using the input parameter table information.

Do not go to the Splunk Add-on for Google Cloud Platform configuration page under Settings > Data Inputs to configure Google Cloud Platform inputs. This page is not supported for this type of input.

Configure Cloud Storage Bucket inputs using configuration file

Follow these steps to configure Cloud Pub/Sub Based Bucket inputs.

1. In your Splunk platform deployment, navigate to $SPLUNK_HOME/etc/apps/Splunk_TA_google-cloudplatform/local.
2. Create a file named inputs.conf, if it does not already exist.
3. Add the following stanza for Cloud Pub/Sub Lite input:

   [google_cloud_pubsub_based_bucket://<input_stanza_name>]
   google_credentials_name = <value>
   google_project = <value>
   google_subscriptions = <value>
   index = <value>
   message_batch_size = <value>
   number_of_threads = <value>
   sourcetype = google:gcp:buckets:data
   

4. Save and restart the Splunk platform.

Input Parameters

Each attribute in the following table corresponds to a field in Splunk Web.

Attribute	Corresponding field in Splunk Web	Description
input_stanza_name	Name	Enter a unique name of the Cloud Pub/Sub Based Bucket input.
google_credentials_name	Credentials	Google Credentials configured using Configuration > Google Credentials or stanza defined in the google_cloud_credentials.conf
google_project	Project	The project from which you want to collect data.
google_subscriptions	Pub/Sub Subscription	The subscription from which you want to pull messages.
index	Index	The index in which to store the Google Cloud Pub/Sub Based Bucket data.
sourcetype	Sourcetype	The sourcetype to use for this input.
message_batch_size	Message Batch Size	Max number of messages to pull from Pub/Sub in one batch. The default value is 10.
number_of_threads	Number of Threads	The number of threads used to collect data in parallel. The dafault value is 10.