Use the Amazon Metadata Connector with Splunk DSP

Use the Amazon Metadata Connector to collect metadata from the resources and infrastructure in Amazon Web Services (AWS).

To use the Amazon Metadata Connector, start by creating a connection that allows it to access data from AWS. Then, add the Amazon Metadata Connector to the start of your data pipeline and configure it to use the connection that you created.

Behavior of the Amazon Metadata Connector

The Amazon Metadata Connector uses AWS regions and AWS APIs to collect resource status and infrastructure information.

All credentials are transmitted securely by HTTPS and saved in the Collect service with industry-standard encryption. They can't be accessed outside of the current tenant.

The connector supports the AWS APIs described in the following table:

AWS API	AWS Permission	Source	Source type	Body
ec2_instances	ec2:DescribeInstances	<region>:ec2:describeInstances	aws:ec2:instance	All attributes of ec2.Instance and OwnerID of ec2.Reservation
ec2_key_pairs	ec2:DescribeKeyPairs	<region>:ec2:describeKeyPairs	aws:ec2:keyPair	All attributes of ec2.KeyPairInfo
ec2_reserved_instances	ec2:DescribeReservedInstances	<region>:ec2:describeReservedInstances	aws:ec2:reservedInstances	All attributes of ec2.ReservedInstances
ebs_snapshots	ec2:DescribeSnapshots	<region>:ec2:describeSnapshots	aws:ec2:snapshot	All attributes of ec2.Snapshot
ec2_volumes	ec2:DescribeVolumes	<region>:ec2:describeVolumes	aws:ec2:volume	All attributes of ec2.Volume
ec2_security_groups	ec2:DescribeSecurityGroups	<region>:ec2:describeSecurityGroups	aws:ec2:securityGroup	All attributes of ec2.SecurityGroup
ec2_images	ec2:DescribeImages	<region>:ec2:describeImages	aws:ec2:image	All attributes of ec2.Image
ec2_addresses	ec2:DescribeAddresses	<region>:ec2:describeAddresses	aws:ec2:address	All attributes of ec2.Address
classic_load_balancers	elasticloadbalancing:DescribeLoadBalancers elasticloadbalancing:DescribeTags elasticloadbalancing:DescribeInstanceHealth	<region>:elb:describeLoadBalancers	aws:elb:loadBalancer	All attributes of elb.LoadBalancerDescription Tags: All attributes of elb.Tags Instances: All attributes of elb.InstanceState
application_load_balancers	elasticloadbalancing:DescribeLoadBalancers elasticloadbalancing:DescribeListeners elasticloadbalancing:DescribeTags elasticloadbalancing:DescribeTargetHealth elasticloadbalancing:DescribeTargetGroups	<region>:elbv2:describeLoadBalancers	aws:elbv2:loadBalancer	All attributes of elbv2.LoadBalance Listeners: All attributes of elbv2.Listeners Tags: All attributes of elbv2.Tags TargetGroups: All attributes of elbv2.TargetGroup and elbv2.TargetHealth
vpcs	ec2:DescribeVpcs	<region>:ec2:describeVpcs	aws:ec2:vpc	All attributes of ec2.Vpc
vpc_subnets	ec2:DescribeSubnets	<region>:ec2:describeSubnets	aws:ec2:subnet	All attributes of ec2.Subnet
vpc_network_acls	ec2:DescribeNetworkAcls	<region>:ec2:describeNetworkAcls	aws:ec2:networkAcl	All attributes of ec2.NetworkAcl
cloudfront_distributions	cloudfront:ListDistributions	<region>:cloudfront:listDistributions	aws:cloudfront:distribution	All attributes of cloudfront.DistributionSummary
rds_instances	rds:DescribeDBInstances	<region>:rds:describeDBInstances	aws:rds:dbInstance	All attributes of rds.DBInstance
lambda_functions	lambda:ListFunctions	<region>:lambda:listFunctions	aws:lambda:function	All attributes of lambda.FunctionConfiguration
s3_buckets	s3:ListAllMyBuckets	<region>:s3:listBuckets	aws:s3:bucket	All attributes of s3.Bucket
iam_users	iam:ListUsers iam:ListAccessKeys iam:GetAccessKeyLastUsed iam:GetAccountPasswordPolicy	<region>:iam:listUsers	aws:iam:user	All attributes of iam.User AccessKey: All attributes of iam.AccessKeyMetadata AccessKey.AccessKeyLastUsed: all attributes of iam.AccessKeyLastUsed PasswordPolicy: All attributes of iam.PasswordPolicy
eks_clusters	eks:DescribeCluster eks:ListClusters	<region>:eks:describeCluster	aws:eks:cluster	All attributes of EKS.ListClusters and EKS.DescribeCluster
route53_domains	route53domains:ListDomains route53domains:GetDomainDetail route53domains:ListTagsForDomain (optional)	<region>:route53Domains:getDomainDetail	aws:route53Domains:domain	All attributes of Route53Domain.ListDomains and Route53Domain.GetDomainDetail
acm_certificates	acm:DescribeCertificate acm:ListCertificates acm:ListTagsForCertificate (optional)	<region>:acm:describeCertificate	aws:acm:certificate	All attributes of ACM.ListCertificates, acm.DescribeCertificate, and acm.ListTagsForCertificate
route53_traffic_policy_instances	route53:ListTrafficPolicyInstances route53:ListTagsForResource (optional)	<region>:route53:listTrafficPolicyInstances	aws:route53:trafficPolicyInstance	All attributes of Route53.ListTrafficPolicyInstances and route53.ListTagsForResource
route53_hosted_zones	route53:ListHostedZones route53:GetHostedZone route53:ListTagsForResource (optional)	<region>:route53:getHostedZone	aws:route53:hostedZone	All attributes of Route53.ListHostedZones, Route53.GetHostedZone. and route53.ListTagsForResource
route53_traffic_policies	route53:ListTrafficPolicies route53:GetTrafficPolicy route53:ListTagsForResource (optional)	<region>:route53:getTrafficPolicy	aws:route53:trafficPolicy	All attributes of Route53.ListTrafficPolicies, Route53.GetTrafficPolicy, and route53.ListTagsForResource
ecr_repositories	ecr:DescribeRepositories	<region>:ecr:describeRepositories	aws:ecr:repository	All attributes of ECR.DescribeRepositories
ecr_images	ecr:DescribeRepositories ecr:DescribeImages	<region>:ecr:describeImages	aws:ecr:image	All attributes of ECR.DescribeRepositories and ECR.DescribeImages
ecs_container_instances	ecs:ListClusters ecs:ListContainerInstances ecs:DescribeContainerInstances	<region>:ecs:describeContainerInstances	aws:ecs:containerInstance	All attributes of ECS.ListClusters, ECS.ListContainerInstances, and ECS.DescribeContainerInstances
ecs_tasks	ecs:ListClusters ecs:ListTasks ecs:DescribeTasks	<region>:ecs:describeTasks	aws:ecs:task	All attributes of ECS.ListClusters, ECS.ListTasks, and ECS.DescribeTasks
ecs_services	ecs:ListClusters ecs:ListServices ecs:DescribeServices	<region>:ecs:describeServices	aws:ecs:service	All attributes of ECS.ListClusters, ECS.ListServices, and ECS.DescribeServices
ecs_clusters	ecs:ListClusters ecs:DescribeClusters	<region>:ecs:describeClusters	aws:ecs:cluster	All attributes of ECS.ListClusters and ECS.DescribeClusters
efs_file_systems	elasticfilesystem:DescribeFileSystems	<region>:efs:describeFileSystems	aws:efs:fileSystem	All attributes of EFS.DescribeFileSystems
dynamodb_tables	dynamodb:ListTables dynamodb:DescribeTable	<region>:dynamoDB:describeTable	aws:dynamoDB:table	All attributes of DynamoDB.ListTables and DynamoDB.DescribeTable
dynamodb_global_tables	dynamodb:ListGlobalTables dynamodb:DescribeGlobalTable	<region>:dynamoDB:describeGlobalTable	aws:dynamoDB:globalTable	All attributes of DynamoDB.ListGlobalTables and DynamoDB.DescribeGlobalTable
waf_web_acls	waf:ListWebACLs waf:GetWebACL	<region>:waf:getWebACL	aws:waf:webACL	All attributes of Waf.ListWebACLs and Waf.GetWebACL
cloudwatchlogs_log_groups	logs:DescribeLogGroups logs:ListTagsLogGroup (optional) logs:GetLogGroupFields (optional)	<region>:cloudwatchlogs:describeLogGroups	aws:cloudwatchlogs:logGroup	All attributes of CloudWatchLogs.DescribeLogGroups, CloudWatchLogs.ListTagsLogGroup, and CloudWatchLogs.GetLogGroupFields

Create a connection using the Amazon Metadata Connector

Create a connection so that the Amazon Metadata Connector can access data from AWS and send the data into a DSP pipeline.

If you are editing a connection that's being used by an active pipeline, you must reactivate that pipeline after making your changes.

Prerequisites

Before you can use the Amazon Metadata Connector, you must have an AWS account. If you do not have an AWS account, ask your AWS administrator to create an account and provide the access key ID and secret access key. Search for "Access Keys (Access Key ID and Secret Access Key)" in the AWS documentation for more information about access key credentials.

Make sure that your AWS account has the necessary permissions for each API that you want to collect data from. If you want to collect data from all the supported APIs, your account needs the following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeReservedInstances",
                "ec2:DescribeSnapshots",
                "ec2:DescribeVolumes",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeImages",
                "ec2:DescribeAddresses",
                "elasticloadbalancing:DescribeLoadBalancers",
                "elasticloadbalancing:DescribeListeners",
                "elasticloadbalancing:DescribeTags",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeInstanceHealth",
                "ec2:DescribeVpcs",
                "ec2:DescribeSubnets",
                "ec2:DescribeNetworkAcls",
                "cloudfront:ListDistributions",
                "rds:DescribeDBInstances",
                "lambda:ListFunctions",
                "s3:ListAllMyBuckets",
                "iam:GetAccountPasswordPolicy",
                "iam:GetAccessKeyLastUsed",
                "iam:ListUsers",
                "iam:ListAccessKeys",
                "eks:DescribeCluster",
                "eks:ListClusters",
                "route53domains:ListDomains",
                "route53domains:GetDomainDetail",
                "route53domains:ListTagsForDomain",
                "acm:DescribeCertificate",
                "acm:ListCertificates",
                "acm:ListTagsForCertificate",
                "route53:ListTrafficPolicyInstances",
                "route53:ListTagsForResource",
                "route53:ListHostedZones",
                "route53:GetHostedZone",
                "route53:ListTagsForResource",
                "route53:ListTrafficPolicies",
                "route53:GetTrafficPolicy",
                "route53:ListTagsForResource",
                "ecr:DescribeRepositories",
                "ecr:DescribeRepositories",
                "ecr:DescribeImages",
                "ecs:ListClusters",
                "ecs:ListContainerInstances",
                "ecs:DescribeContainerInstances",
                "ecs:ListClusters",
                "ecs:ListTasks",
                "ecs:DescribeTasks",
                "ecs:ListClusters",
                "ecs:ListServices",
                "ecs:DescribeServices",
                "ecs:ListClusters",
                "ecs:DescribeClusters",
                "elasticfilesystem:DescribeFileSystems",
                "dynamodb:ListTables",
                "dynamodb:DescribeTable",
                "dynamodb:ListGlobalTables",
                "dynamodb:DescribeGlobalTable",
                "waf:ListWebACLs",
                "waf:GetWebACL",
                "logs:DescribeLogGroups",
                "logs:ListTagsLogGroup",
                "logs:GetLogGroupFields"
            ],
            "Resource": "*"
        }
    ]
}

If you want to collect data from a subset of the supported AWS APIs, you only need to add the permissions for those particular APIs.

Steps

1. From the Data Management page, click the Connections tab.
2. Click Create New Connection.
3. Select Amazon Metadata Connector and then click Next.
4. Complete the following fields:

   Field	Description
   Connection Name	A unique name for your connection.
   Access Key ID	Your AWS access key ID.
   Secret Access Key	Your AWS secret access key.
   Region API Groups	A list of groups that indicate which combinations of regions and APIs the connector collects data from. For each group that you want to define, click Add Group and select the appropriate values from the following drop-down lists: Regions: A list of regions that you want to collect data from. APIs (Optional): If you don't want to collect data from all the supported APIs, type a list of the specific APIs that you want to collect data from.
   Scheduled	This parameter is on by default, indicating that jobs run automatically. Toggle this parameter off to stop the scheduled job from automatically running. Jobs that are currently running are not affected.
   Schedule	The time-based job schedule that determines when the connector executes jobs for collecting data. Select a predefined value or write a custom CRON schedule. All CRON schedules are based on UTC.
   Workers	The number of workers you want to use to collect data.

   If your data fails to get into DSP, check the fields again to make sure you have the correct name, AWS access key ID, AWS secret access key, and region API groups for your Amazon Metadata connection. DSP doesn't run a check to see if you enter the valid credentials.

5. Click Save.

You can now use your connection in a data pipeline.