Connect to AWS and send data to Splunk Observability Cloud 🔗
To leverage the benefits of data monitoring across your infrastructure, connect Splunk Observability Cloud to Amazon Web Services (AWS). Follow these steps:
Verify the prerequisites.
Evaluate your needs and learn about Observability Cloud’s options to ingest AWS data.
Choose how to connect with AWS.
Check the list of AWS integrations available in Splunk Observability Cloud.
To connect AWS to Observability Cloud you need:
Administrator privileges in Observability Cloud and your AWS accounts.
An authentication method.
Observability Cloud supports the following regions:
ap-northeast-1: Asia Pacific (Tokyo)
ap-northeast-2: Asia Pacific (Seoul)
ap-northeast-3: Asia Pacific (Osaka)
ap-south-1: Asia Pacific (Mumbai)
ap-southeast-1: Asia Pacific (Singapore)
ap-southeast-2: Asia Pacific (Sydney)
ca-central-1: Canada (Central)
eu-central-1: Europe (Frankfurt)
eu-north-1: Europe (Stockholm)
eu-west-1: Europe (Ireland)
eu-west-2: Europe (London)
eu-west-3: Europe (Paris)
sa-east-1: South America (Sao Paulo)
us-east-1: US East (N. Virginia)
us-east-2: US East (Ohio)
us-west-1: US West (N. California)
us-west-2: US West (Oregon)
af-south-1: Africa (Cape Town)
ap-east-1: Asia Pacific (Hong Kong)
ap-south-2: Asia Pacific (Hyderabad)
ap-southeast-3: Asia Pacific (Jakarta)
ap-southeast-4: Asia Pacific (Melbourne)
eu-central-2: Europe (Zurich)
eu-south-1: Europe (Milan)
eu-south-2: Europe (Spain)
me-central-1: Middle East (UAE)
me-south-1: Middle East (Bahrain)
us-gov-east-1: AWS GovCloud (US-East)
us-gov-west-1: AWS GovCloud (US-West)
cn-north-1: China (Beijing)
cn-northwest-1: China (Ningxia)
If you want to activate a specific optional region, you need to do it before adding it to the integration. Make sure you’ve activated the optional regions you’ll need in your AWS console first. Regular regions are activated in AWS by default.
If you’re using the UI guided setup to create the integration, you’ll be prompted to select which AWS regions you work with.
If you’re using the API and supply an empty list in an API call, Observability Cloud activates all regular regions. If you add the
ec2:DescribeRegionspermission to your AWS policy, optional regions you’ve activated on your AWS account are activated in Observability Cloud as well.
In most AWS regions, use an Identity and Access Management (IAM) policy, an AWS IAM role, and an external ID from Observability Cloud.
An external ID is a random string used to establish a trust relationship between Observability Cloud and your AWS account. It’s automatically generated for you when you create a new AWS integration in Observability Cloud. See How to use an external ID when granting access to your AWS resources to a third party in AWS documentation.
For the GovCloud or China regions, select the option to authenticate using a secure token, which combines an access key ID and a secret access key.
The AWS IAM policy is a JSON object to which Observability Cloud refers for permission to collect data from every supported AWS service. To create a new AWS IAM policy, follow these steps.
Log into your Amazon Web Services account and look for the Identity and Access Management service.
Create a new policy. In the JSON tab, replace the placeholder JSON with the pertinent AWS IAM policy JSON. Guided setup provides this policy in the Prepare AWS Account step. See also some policy examples.
Follow the instructions to complete the process and create the policy.
If you have any doubts, check AWS documentation.
After creating an AWS IAM policy, you need to assign that policy to a particular role by performing the following steps in the Amazon Web Services console:
Go to Roles > Create Role and select Another AWS account as the type of trusted entity.
Copy and paste the Account ID displayed in guided setup into the Account ID field.
Select Require external ID. Copy and paste the External ID displayed in the guided setup into the External ID field.
Continue with Next: Permissions. Under Policy name, select the policy you made in the previous step.
Follow the instructions, and name and create your new AWS IAM role.
Creating the AWS IAM role generates the
Role ARN used to establish connection with AWS. Copy the created ARN role, and paste it into the Role ARN field in the guided setup.
If you have any doubts, check AWS documentation.
Regardless of the connection option you choose, you can configure your system more efficiently if you decide beforehand what data types and sources you want.
To determine the best connection method and configuration settings, answer the following questions before you connect AWS to Splunk Observability Cloud:
Which AWS regions do you need to work with.
Do you want to collect metrics through API polling at specified intervals, or through CloudWatch Metric Streams? You can activate Metric Streams using the Splunk Observability Cloud API.
If you want to collect logs in addition to metrics, include logs while configuring the API or when given that option while performing a guided setup.
You can poll data from AWS at specified intervals using CloudWatch APIs. Due to the CloudWatch metrics instability, for certain namespaces some metrics might be delayed a few minutes. See more in Configure API polling.
After you create an AWS integration, if more than 100,000 metrics are retrieved from CloudWatch, Observability Cloud automatically deactivates the integration and sends you a warning email.
This check runs just once per integration. If you activate the integration afterwards, it will work correctly.
You can deactivate this check by setting the
enableCheckLargeVolume field in the AWS integration to
false using the API.
If you filter data based on tags, your costs for Amazon CloudWatch and Splunk Infrastructure Monitoring might decrease.
Be careful when choosing tag names: Splunk Observability Cloud only allows alphanumeric characters, and the underscore and minus symbols. Unsupported characters include
@, and spaces, which are replaced by the underscore character.
CloudWatch Metric Streams doesn’t support filtering based on resource tags.
Rather than polling for metrics data, CloudWatch Metric Streams sends metrics to a Kinesis Data Firehose stream, reducing latency. See Low Latency Observability Into AWS Services With Splunk in the DevOps blog for more information.
CloudWatch Metric Streams continually stream Amazon CloudWatch metrics. Although they’re more efficient than API polling, consider the constraints below.
In most cases, metrics are reported every minute. However, some services use a different cadence: For example, selected S3 metrics are reported on a daily basis. Check AWS documentation to verify how often your services’ metrics are reported.
Collecting Amazon CloudWatch metrics via the polling APIs at the default polling rate of 300 seconds (5 minutes) is generally cheaper than using Metric Streams. On the other hand, if you set polling intervals to one minute, generally you’ll see an increase in Amazon CloudWatch usage costs compared to Metric Streams.
Learn more at Amazon CloudWatch usage costs.
You can connect Observability Cloud to AWS in several ways. By default, Observability Cloud brings in data from all supported AWS services associated with your account. To limit the amount of data to import, see Specify and limit the data and metadata to import.
Choose the connection method that best matches your needs:
Why use this?
Connect to AWS using the guided setup in Splunk Observability Cloud
Guides you step-by-step to set up an AWS connection and default configuration in Observability Cloud. Guided setup includes links to Amazon CloudFormation templates that you can select to create needed AWS IAM roles.
Connect to AWS using the Splunk Observability Cloud API
Requires knowledge of POST and PUT call syntax, but includes options and automation that are not part of the guided setup. Choose this method if you want to configure many integrations at once.
Connect to AWS using Splunk Terraform
Use this connection method if you already manage your infrastructure as code by deploying through Terraform.
See also the Splunk add-on for Amazon Kinesis Firehose.
If you can’t connect AWS to Observability Cloud, see Troubleshoot your AWS connection.
Splunk is not responsible for data availability, and it can take up to several minutes (or longer, depending on your configuration) from the time you connect until you start seeing valid data from your account.”
See Leverage data from integration with AWS for an overview of what you can do after you connect Observability Cloud to AWS.
Learn about our AWS Infrastructure Monitoring options. You’ll find instructions on how to import AWS metrics and metadata, or AWS tag and log information using namespaces and filters.
Refer to the AWS official documentation for a list of the available AWS metrics and other data, or read about the metadata we provide.
To collect traces and metrics of your AWS Lambda functions for Splunk APM, see Instrument AWS Lambda functions for Splunk Observability Cloud.