Splunk® Data Stream Processor

Connect to Data Sources and Destinations with DSP

On April 3, 2023, Splunk Data Stream Processor reached its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.

All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.
This documentation does not apply to the most recent version of Splunk® Data Stream Processor. For documentation on the most recent version, go to the latest release.

Connecting Google Cloud Monitoring to your DSP pipeline

The Google Cloud Monitoring connector is planned for deprecation. See the Release Notes for more information.

When creating a data pipeline in Splunk Data Stream Processor, you can connect to Google Cloud Monitoring and use it as a data source. Google Cloud Monitoring helps you understand the performance, uptime, and health of your cloud-powered applications. You can get metrics data from Google Cloud Monitoring into a pipeline, transform the data as needed, and then send the transformed data out from the pipeline to a destination of your choosing.

To connect to Google Cloud Monitoring as a data source, you must complete the following tasks:

  1. Create a connection that allows DSP to access your Google Cloud Monitoring data. See Create a DSP connection to Google Cloud Monitoring.
  2. Create a pipeline that starts with the Google Cloud Monitoring source function. See the Building a pipeline chapter in the Use the Data Stream Processor manual for instructions on how to build a data pipeline.
  3. Configure the Google Cloud Monitoring source function to use your Google Cloud Monitoring connection. See Get data from Google Cloud Monitoring in the Function Reference manual.

When you activate the pipeline, the source function starts collecting metrics from Google Cloud Monitoring. Each metric event is received into the pipeline as a record.

If your data fails to get into DSP, check the connection settings to make sure you have the correct private key ID, private key, and client email for your Google Cloud Monitoring account, as well as the correct project IDs. DSP doesn't run a check to see if you enter valid credentials.

How Google Cloud Monitoring data is collected

The source function collects data according to the job schedule that you specified in the connection settings. See Scheduled data collection jobs for more information, including a list of the limitations that apply to all scheduled data collection jobs.

The following behavior from the Google Cloud Monitoring connector might have an impact on the exact timing and scope of your data collection jobs:

  • The connector uses the Google Cloud Monitoring API to collect metrics. If your API quota is exceeded, the connector sleeps for 60 seconds and then retries the API calls. The connector sleeps and retries a maximum of 5 times. You can request higher Monitoring API limits using the Google Cloud Console. Search for "Monitoring API quotas and limits" in the Google Cloud Monitoring documentation.
  • The first execution of a job sets the current time as the start time for collecting data. No data is collected in the first execution of a job. The following job executions collect all the data points from the last job execution up to the current time.
  • The connector collects data from all supported metric types if no metric types are specified in the connection settings.
  • A job can collect data from up to a maximum of 20,000 metric types per execution. Google Cloud Monitoring has approximately 3200 default metric types per project. If you are collecting data from all supported metric types, metrics data from a maximum of 6 projects can be collected per job execution. Use the Metric Types parameter in the connection settings to filter out metric types that are not relevant to the metrics data you are monitoring.

Performance of the Google Cloud Monitoring connector

A connector with 5 workers can collect metrics from 500 to 600 metric types per minute. It takes approximately 5 minutes to collect all metric types from 1 project.

Limitations of the Google Cloud Monitoring connector

The Google Cloud Monitoring connector has the following limitations:

  • The connector supports bool, double, and int64 data types. All other data types are not supported.
  • DEPRECATED and EARLY_ACCESS metrics are skipped.
Last modified on 29 March, 2022
Create a DSP connection to Apache Pulsar   Create a DSP connection to Google Cloud Monitoring

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.0, 1.2.1-patch02, 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5, 1.3.0, 1.3.1

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters