Splunk® Data Stream Processor

Getting Data In

Acrobat logo Download manual as PDF


This documentation does not apply to the most recent version of DSP. Click here for the latest version.
Acrobat logo Download topic as PDF

Use the Google Cloud Monitoring Metrics Connector with Splunk DSP

Use the Google Cloud Monitoring Metrics Connector to collect metrics data from Google Cloud Monitoring. Cloud Monitoring helps you understand the performance, uptime, and health of your cloud-powered applications. Search for "Cloud Monitoring" in the Google documentation for more information about setting up and using Cloud Monitoring.

To use the Google Cloud Monitoring Metrics Connector, start by creating a connection that allows it to access data from Google Cloud Monitoring. Then, add the Google Cloud Monitoring Metrics Connector to the start of your data pipeline and configure it to use the connection that you created.

Behavior of the Google Cloud Monitoring Metrics Connector

The Google Cloud Monitoring Metrics Connector has the following behavior:

  • The connector uses the Google Cloud Monitoring API to collect metrics. If your API quota is exceeded. the connector sleeps for 60 seconds and then retries the API calls. The connector sleeps and retries a maximum of 5 times. You can request higher Monitoring API limits using the Google Cloud Console. Search for "Monitoring API quotas and limits" in the Google Cloud Monitoring documentation.
  • The first execution of a job sets the current time as the start time for collecting data. No data is collected in the first execution of a job. The following job executions collect all the data points from the last job execution up to the current time.
  • The connector collects data from all supported metric types if no metric types are specified in the parameters.
  • A job can collect data from up to a maximum of 20,000 metric types per execution. Google Cloud Monitoring has approximately 3200 default metric types per project. If you are collecting data from all supported metric types, metrics data from a maximum of 6 projects can be collected per job execution. Use the Metric Types parameter to filter out metric types that are not relevant to the metrics data you are monitoring.

Performance of the Google Cloud Monitoring Metrics Connector

A connector with 5 workers can collect metrics from 500 to 600 metric types per minute. It takes approximately 5 minutes to collect all metric types from 1 project.

Limitations of the Google Cloud Monitoring Metrics Connector

The Google Cloud Monitoring Metrics Connector has the following limitations:

  • The connector supports bool, double, and int64 data types. All other data types are not supported.
  • DEPRECATED and EARLY_ACCESS metrics are skipped.

Create a connection using the Google Cloud Monitoring Metrics Connector

Created a connection so that the Google Cloud Monitoring metrics Connector can access data from Google Cloud Monitoring and send the data into a DSP pipeline.

If you are editing a connection that's being used by an active pipeline, you must reactivate that pipeline after making your changes.

Prerequisites

Before you can use the Google Cloud Monitoring Metrics Connector you must have a Google Cloud Monitoring account. If you don't have a Google Cloud Monitoring account, ask your admin to create an account and provide the private key ID, the private key, and the client email. Search for "Creating and managing service account keys" in the Google Cloud Monitoring documentation for more information about creating and managing service account keys using the Google Cloud Console.

Make sure your Google Cloud Monitoring account has at least the following permissions:

  • metricDescriptors.list
  • timeSeries.list

You can grant these permissions using the monitoring.Viewer role.

Steps

  1. From the Data Management page, click on the Connections tab.
  2. Click Create New Connection.
  3. Choose the Google Cloud Monitoring Metrics Connector.
  4. Click Next.
  5. Complete the following fields:
    Field Description
    Connection Name A unique name for your connection.
    Private Key ID Your Google Cloud Monitoring account private key ID.
    Private Key Your Google Cloud Monitoring account private key.
    Client Email The client email address linked to your Google Cloud Monitoring account.
    Project IDs A list of project IDs.
    Metric Types (Optional) A list of metric types that you want to collect data from. The list is used as a prefix to filter the metric types.
    Scheduled This parameter is on by default, indicating that jobs run automatically. Toggle this parameter off to stop the scheduled job from automatically running. Jobs that are currently running are not affected.
    Schedule The time-based job schedule that determines when the connector executes jobs for collecting data. Select a predefined value or write a custom CRON schedule. All CRON schedules are based on UTC.
    Workers The number of workers you want to use to collect data.

    If your data fails to get into DSP, check the fields again to make sure you have the correct name, private key ID, private key, client email, and project IDs for your Google Cloud Monitoring Metrics connection. DSP doesn't run a check to see if you enter the valid credentials.

  6. Click Save.

You can now use your connection in a connector in a data pipeline.

Last modified on 23 October, 2020
PREVIOUS
Use the Azure Monitor Metrics Connector with Splunk DSP
  NEXT
Use the Microsoft 365 Connector with Splunk DSP

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.1.0


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters