Splunk® Data Stream Processor

Getting Data In

On April 3, 2023, Splunk Data Stream Processor will reach its end of sale, and will reach its end of life on February 28, 2025. If you are an existing DSP customer, please reach out to your account team for more information.
This documentation does not apply to the most recent version of Splunk® Data Stream Processor. For documentation on the most recent version, go to the latest release.

Use the Azure Monitor Metrics connector with Splunk DSP

Use the Azure Monitor Metrics connector to collect metrics data from Azure Monitor. Azure Monitor helps you understand how your applications are performing, and proactively identifies issues affecting them and the resources they depend on. See Azure Monitor overview in the Microsoft Azure documentation for more information about setting up and using Azure Monitor. See also Deserialize and send Azure Event Hubs data from a DSP pipeline.

Prerequisites

Before you can use the Azure Monitor Metrics connector, you must create an integration application in Azure AD. For more information on creating integration applications, see How to: Use the portal to create an Azure AD application and service principal that can access resources in the Microsoft Azure documentation.

When creating your application in Azure AD, make sure you create a client secret and make a note of the following parameters:

  • Directory ID (tenant_id)
  • Application ID (client_id)
  • Client Secret (client_secret)

Permissions for the Azure Monitor Metrics connector

Make sure your AD application has the Reader role.

Parameters used in the Azure Monitor Metrics connector

In addition to the common configuration parameters, the Azure Monitor Metrics connector uses the following parameters:

  • ms_profile: The Microsoft profile used to access the Azure Monitor Metrics
    • tenant_id: The Directory ID from Azure Active Directory
    • client_id: The Application ID from the registered application within the Azure Active Directory
    • client_secret: The registered application key for the corresponding application
  • subscription: The Azure subscription you want to collect metrics for
  • locations: (Optional) An array of Azure locations you want to collect metrics for. Location names must be entered without spaces and all in lowercase. For example, "East US 2" is eastus2. The Azure Monitor Metrics connector collects data for all locations by default. See Azure locations in the Microsoft Azure documentation for a complete list of supported locations.
  • resource_groups: (Optional) An array of Azure resource groups you want to collect metrics for. The Azure Monitor Metrics connector collects data for all resource groups by default.
  • metrics: (Optional) An array of resource types. Resource types must be entered as <resource_provider>/<type>. See Supported metrics with Azure Monitor in the Microsoft Azure documentation for a list of supported metrics.
  • aggregations: (Optional) An array of aggregations to fetch for a metric data-point, for example, ["Average", "Maximum"]. Valid Azure aggregations are Average, Count, Maximum, Minimum, and Total>. The Azure Monitor Metrics Connector will fetch all metric-supported aggregations by default.
  • delay: (Optional) The number of minutes to delay the metrics data collection to compensate for latency in the availability of Azure Monitor Metrics data points. The default is 5 minutes.

Behavior of the Azure Monitor Metrics connector

The Azure Monitor Metrics connector has the following behavior:

  • The metrics are collected for the finest time-grain available in the Azure metric definition.
  • The first time a scheduled job runs, metrics are collected for only one time interval. For all following scheduled jobs, metrics are collected from the last execution up to the current time, minus the number of minutes defined by the delay parameter.

Azure Monitor Metrics connector output

The following dimensions are added to the metric events collected from Azure Monitor:

  • SubscriptionId: The subscription ID of the Azure account
  • TenantId: The directory ID from Azure Active Directory
  • ResourceId: The resource ID of the metric the event belongs to
  • Location: The location of the resource
  • Interval: The time granularity of the Azure metric expressed as an ISO 8601 duration (for example PT1H)

A typical metric event looks like this: 

{
    "metric_name": "Microsoft.Compute/virtualMachines.PercentageCPU.Average",
    "_time": 2019-09-30T01:00:00.00+00:00,
    "_value": 0.7575,
    "unit": "Percent",
    "Location": "westus",                                           //dimension
    "TenantId": "aa1bb2c3-cccc-4444-5555-12a3b4567ab0",             //dimension
    "SubscriptionId": "aa1bb2c3-cccc-4444-5555-12a3b4567ab0",       //dimension
    "ResourceId": "/subscriptions/aa1bb2c3-cccc-4444-5555-12a3b4567ab0/resourceGroups/exampleResourceGroup/providers/Microsoft.Compute/virtualMachines/vm_name",        //dimension
    "Interval": "PT1M",                                              //dimension
    "source": "westus:Microsoft.Compute/virtualMachines",
    "host": "test-local",
    "index": "metrics",
    "sourcetype": "azure:monitor:metrics",
}

Create, modify, and delete a scheduled job using the Collect API

You can create, modify, and delete a scheduled job in the Azure Monitor Metrics connector using the Collect API.

Create a scheduled job

The following example creates a job, schedules it to run at 45 minutes past every hour, and assigns 2 workers. 

curl -X POST "https://<DSP_HOST>:31000/default/collect/v1beta1/jobs/" \
    -H "Authorization: Bearer <accessToken>" \
    -H "Content-Type: application/json" \
    -d '{
            "name": "azure-monitor-metrics-job",
            "connectorID": "azure-monitor-metrics",
            "schedule": "45 * * * *",
            "parameters": {
                "ms_profile": {
                    "tenant_id": "your tenant ID",
                    "client_id": "your client ID",
                    "client_secret": "your client secret"
                },
                "subscription": "your subscription ID",
                "metrics": ["Microsoft.Compute/virtualMachines"],
                "aggregations": ["Average", "Maximum"]
            },
            "enabled": true,
            "scalePolicy": {
                "static": {
                    "workers": 2
                }
            }
        }'

A typical response when you create a scheduled job using a POST request looks like this: 

{
    "data": {
        "connectorID": "azure-monitor-metrics",
        "createUserID": "your user ID",
        "createdAt": "2019-09-30T21:08:44.083Z",
        "id": "your job ID",
        "lastModifiedAt": "2019-09-30T21:08:44.083Z",
        "lastUpdateUserID": "last user who updated",
        "name": "your connection name",
        "schedule": "45 * * * *",
        "scheduled": true,
        "tenant": "your tenant ID",
        "eventExtraFields": null,
        "parameters": {
            "aggregations": [
                "Average",
                "Maximum"
            ],
            "metrics": [
                "Microsoft.Compute/virtualMachines"
            ],
            "ms_profile": {
                "client_id": "your client ID",
                "tenant_id": "your tenant ID"
            },
            "subscription": "your subscription ID"
        },
        "scalePolicy": {
            "static": {
                "workers": 2
            }
        }
    }
}

Verify the job

After you create the scheduled job, you can find the job id in the POST response. The following example performs a GET request on the job id to verify the job was created and scheduled correctly: 

curl -X GET "https://<DSP_HOST>:31000/default/collect/v1beta1/jobs/<jobId>" \
    -H "Authorization: Bearer <accessToken>" \
    -H "Content-Type: application/json"

A typical response for a GET request on a job id in a scheduled job is the same as a POST response and looks like this: 

{
    "data": {
        "connectorID": "azure-monitor-metrics",
        "createUserID": "your user ID",
        "createdAt": "2019-09-30T21:08:44.083Z",
        "id": "your job ID",
        "lastModifiedAt": "2019-09-30T21:08:44.083Z",
        "lastUpdateUserID": "last user who updated",
        "name": "your connection name",
        "schedule": "45 * * * *",
        "scheduled": true,
        "tenant": "your tenant ID",
        "eventExtraFields": null,
        "parameters": {
            "aggregations": [
                "Average",
                "Maximum"
            ],
            "metrics": [
                "Microsoft.Compute/virtualMachines"
            ],
            "ms_profile": {
                "client_id": "your client ID",
                "tenant_id": "your tenant ID"
            },
            "subscription": "your subscription ID"
        },
        "scalePolicy": {
            "static": {
                "workers": 2
            }
        }
    }
}

Modify a scheduled job

The following example modifies the scheduled job with the PATCH request to increase the number of workers to 4: 

curl -X PATCH "https://<DSP_HOST>:31000/default/collect/v1beta1/jobs/<jobId>" \
    -H "Authorization: Bearer <accessToken>" \
    -H "Content-Type: application/merge-patch+json" \
    -d '{
            "scalePolicy": {
                "static": {
                    "workers": 4
                }
            }
        }'

A typical response for a PATCH request on a scheduled job looks like this: 

{
    "data": {
        "connectorID": "azure-monitor-metrics",
        "createUserID": "your user ID",
        "createdAt": "2019-09-30T21:08:44.083Z",
        "id": "your job ID",
        "lastModifiedAt": "2019-09-30T21:16:59.817Z",
        "lastUpdateUserID": "last user who updated",
        "name": "your connection name",
        "schedule": "45 * * * *",
        "scheduled": true,
        "tenant": "your tenant ID",
        "eventExtraFields": null,
        "parameters": {
            "aggregations": [
                "Average",
                "Maximum"
            ],
            "metrics": [
                "Microsoft.Compute/virtualMachines"
            ],
            "ms_profile": {
                "client_id": "your client ID",
                "tenant_id": "your tenant ID"
            },
            "subscription": "your subscription ID"
        },
        "scalePolicy": {
            "static": {
                "workers": 4
            }
        }
    }
}

Note that the lastModifiedAt value is automatically updated.

Delete a scheduled job

The following example deletes a scheduled job based on job id: 

curl -X DELETE "https://<DSP_HOST>:31000/default/collect/v1beta1/jobs/<jobId>" \
    -H "Authorization: Bearer <accessToken>" \
    -H "Content-Type: application/json"

When the connection is successfully deleted, you receive a "204 No content" response.

Last modified on 05 March, 2020
Use the AWS Metadata connector with Splunk DSP  

This documentation applies to the following versions of Splunk® Data Stream Processor: 1.0.1

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters