Use the Google Cloud Monitoring Metrics Connector with Splunk DSP

Use the Google Cloud Monitoring Metrics Connector to collect metrics data from Google Cloud Monitoring. Cloud Monitoring helps you understand the performance, uptime, and health of your cloud-powered applications. Search for "Cloud Monitoring" in the Google documentation for more information about setting up and using Cloud Monitoring.

To use the Google Cloud Monitoring Metrics Connector, start by creating a connection that allows it to access data from Google Cloud Monitoring. Then, add the Google Cloud Monitoring Metrics Connector to the start of your data pipeline and configure it to use the connection that you created.

Behavior of the Google Cloud Monitoring Metrics Connector

The Google Cloud Monitoring Metrics Connector has the following behavior:

• The connector uses the Google Cloud Monitoring API to collect metrics. If your API quota is exceeded. the connector sleeps for 60 seconds and then retries the API calls. The connector sleeps and retries a maximum of 5 times. You can request higher Monitoring API limits using the Google Cloud Console. Search for "Monitoring API quotas and limits" in the Google Cloud Monitoring documentation.
• The first execution of a job sets the current time as the start time for collecting data. No data is collected in the first execution of a job. The following job executions collect all the data points from the last job execution up to the current time.
• The connector collects data from all supported metric types if no metric types are specified in the parameters.
• A job can collect data from up to a maximum of 20,000 metric types per execution. Google Cloud Monitoring has approximately 3200 default metric types per project. If you are collecting data from all supported metric types, metrics data from a maximum of 6 projects can be collected per job execution. Use the Metric Types parameter to filter out metric types that are not relevant to the metrics data you are monitoring.

Performance of the Google Cloud Monitoring Metrics Connector

A connector with 5 workers can collect metrics from 500 to 600 metric types per minute. It takes approximately 5 minutes to collect all metric types from 1 project.

Limitations of the Google Cloud Monitoring Metrics Connector

The Google Cloud Monitoring Metrics Connector has the following limitations:

• The connector supports bool, double, and int64 data types. All other data types are not supported.
• DEPRECATED and EARLY_ACCESS metrics are skipped.

Create a connection using the Google Cloud Monitoring Metrics Connector

Created a connection so that the Google Cloud Monitoring metrics Connector can access data from Google Cloud Monitoring and send the data into a DSP pipeline.

If you are editing a connection that's being used by an active pipeline, you must reactivate that pipeline after making your changes.

Prerequisites

Before you can use the Google Cloud Monitoring Metrics Connector you must have a Google Cloud Monitoring account. If you don't have a Google Cloud Monitoring account, ask your admin to create an account and provide the private key ID, the private key, and the client email. Search for "Creating and managing service account keys" in the Google Cloud Monitoring documentation for more information about creating and managing service account keys using the Google Cloud Console.

Make sure your Google Cloud Monitoring account has at least the following permissions:

• metricDescriptors.list
• timeSeries.list

You can grant these permissions using the monitoring.Viewer role.

Steps

1. From the Data Management page, click on the Connections tab.
2. Click Create New Connection.
3. Choose the Google Cloud Monitoring Metrics Connector.
4. Click Next.
5. Complete the following fields:

   Field	Description
   Connection Name	A unique name for your connection.
   Private Key ID	Your Google Cloud Monitoring account private key ID.
   Private Key	Your Google Cloud Monitoring account private key.
   Client Email	The client email address linked to your Google Cloud Monitoring account.
   Project IDs	A list of project IDs.
   Metric Types	(Optional) A list of metric types that you want to collect data from. The list is used as a prefix to filter the metric types.
   Scheduled	This parameter is on by default, indicating that jobs run automatically. Toggle this parameter off to stop the scheduled job from automatically running. Jobs that are currently running are not affected.
   Schedule	The time-based job schedule that determines when the connector executes jobs for collecting data. Select a predefined value or write a custom CRON schedule. All CRON schedules are based on UTC.
   Workers	The number of workers you want to use to collect data.

   If your data fails to get into DSP, check the fields again to make sure you have the correct name, private key ID, private key, client email, and project IDs for your Google Cloud Monitoring Metrics connection. DSP doesn't run a check to see if you enter the valid credentials.

6. Click Save.

You can now use your connection in a connector in a data pipeline.