Connecting Google Cloud Monitoring to your DSP pipeline

The Google Cloud Monitoring connector is planned for deprecation. See the Release Notes for more information.

When creating a data pipeline in Splunk Data Stream Processor, you can connect to Google Cloud Monitoring and use it as a data source. Google Cloud Monitoring helps you understand the performance, uptime, and health of your cloud-powered applications. You can get metrics data from Google Cloud Monitoring into a pipeline, transform the data as needed, and then send the transformed data out from the pipeline to a destination of your choosing.

To connect to Google Cloud Monitoring as a data source, you must complete the following tasks:

1. Create a connection that allows DSP to access your Google Cloud Monitoring data. See Create a DSP connection to Google Cloud Monitoring.
2. Create a pipeline that starts with the Google Cloud Monitoring source function. See the Building a pipeline chapter in the Use the Data Stream Processor manual for instructions on how to build a data pipeline.
3. Configure the Google Cloud Monitoring source function to use your Google Cloud Monitoring connection. See Get data from Google Cloud Monitoring in the Function Reference manual.

When you activate the pipeline, the source function starts collecting metrics from Google Cloud Monitoring. Each metric event is received into the pipeline as a record.

If your data fails to get into DSP, check the connection settings to make sure you have the correct private key ID, private key, and client email for your Google Cloud Monitoring account, as well as the correct project IDs. DSP doesn't run a check to see if you enter valid credentials.

How Google Cloud Monitoring data is collected

The source function collects data according to the job schedule that you specified in the connection settings. See Scheduled data collection jobs for more information, including a list of the limitations that apply to all scheduled data collection jobs.

The following behavior from the Google Cloud Monitoring connector might have an impact on the exact timing and scope of your data collection jobs:

• The connector uses the Google Cloud Monitoring API to collect metrics. If your API quota is exceeded, the connector sleeps for 60 seconds and then retries the API calls. The connector sleeps and retries a maximum of 5 times. You can request higher Monitoring API limits using the Google Cloud Console. Search for "Monitoring API quotas and limits" in the Google Cloud Monitoring documentation.
• The first execution of a job sets the current time as the start time for collecting data. No data is collected in the first execution of a job. The following job executions collect all the data points from the last job execution up to the current time.
• The connector collects data from all supported metric types if no metric types are specified in the connection settings.
• A job can collect data from up to a maximum of 20,000 metric types per execution. Google Cloud Monitoring has approximately 3200 default metric types per project. If you are collecting data from all supported metric types, metrics data from a maximum of 6 projects can be collected per job execution. Use the Metric Types parameter in the connection settings to filter out metric types that are not relevant to the metrics data you are monitoring.

Performance of the Google Cloud Monitoring connector

A connector with 5 workers can collect metrics from 500 to 600 metric types per minute. It takes approximately 5 minutes to collect all metric types from 1 project.

Limitations of the Google Cloud Monitoring connector

The Google Cloud Monitoring connector has the following limitations:

• The connector supports bool, double, and int64 data types. All other data types are not supported.
• DEPRECATED and EARLY_ACCESS metrics are skipped.