Troubleshoot your AWS connection đź”—
If you experience issues when connecting Splunk Observability Cloud to your Amazon Web Services (AWS) account, they might be caused by conflicts between policies and permissions. See Troubleshoot log collection in AWS for specific log troubleshooting and Troubleshoot Metric Streams in AWS for issues specific to Metric Streams.
Caution
Splunk is not responsible for data availability, and it can take up to several minutes (or longer, depending on your configuration) from the time you connect until you start seeing valid data from your account.
Error validating AWS connection đź”—
The automatic attempt to validate a connection that you just configured fails, so there is no connection between Splunk Observability Cloud and your AWS account.
Cause đź”—
The connection might fail due to mismatched Identity Access Management (IAM) policies. To diagnose connection failure, check the permissions or policies you set up and compare them to the permissions that AWS requires.
Verify whether your error message looks similar to this example:
Error validating AWS / Cloudwatch credentials
Validation failed for following region(s):
us-east-1
[ec2] software.amazon.awssdk.services.ec2.model.Ec2Exception: You are not authorized to perform this operation.
If you receive a similar error message, then the IAM policy that you created to connect AWS to Splunk Observability Cloud does not match the policy already in your AWS account.
Similarly, if your AWS account uses a service control policy (SCP) or administrative features such as PermissionsBoundary, then there might be limits on which calls can be made in your organization, even if those calls are covered by your AWS IAM policy.
Solution đź”—
Splunk Observability Cloud uses the following calls to validate whether it can accept data from the AWS Compute Optimizer tool to support CloudWatch metric streams:
client.describeInstanceStatus(),
client.describeTags(),
client.describeReservedInstances(),
client.describeReservedInstancesModifications()
client.describeOrganization()
To ensure that your AWS integration works as expected, revisit your configuration choices in Splunk Observability Cloud to verify that they match the permissions policy in your AWS management console.
A match ensures that conflicting permissions do not cause your AWS environment to block integrations. See the “Amazon CloudWatch permissions reference” in the Amazon documentation for details about the available permissions.
Splunk Observability Cloud doesn’t work as expected 🔗
Features or tools within Splunk Observability Cloud do not work as expected.
Cause đź”—
When a feature in Splunk Observability Cloud does not work as expected after connection to AWS, then permissions for that feature in the AWS IAM policy are absent or blocking implementation. For example, ec2:DescribeRegions is used to detect which AWS regions are active in your account. Without that permission, or if no region is specified, then system settings default to AWS standard regions.
Metrics collection also depends on the the permissions you set.
Solution đź”—
Review your IAM policy to ensure it includes the permissions needed for the metrics or other data that you intend to collect.
Once integrated with your Amazon Web Services account, Splunk Observability Cloud can gather CloudWatch metrics, CloudWatch logs, CloudWatch Metric Streams, service logs stored in Amazon S3 buckets, and service tag and property information. But leveraging the full power of the integration requires all included permissions.
Status check metrics are missing đź”—
Metrics related to status check are missing.
Cause đź”—
By default, status check metrics are not activated to reduce AWS CloudWatch cost and Splunk Observability Cloud system usage.
Solution đź”—
Activate status metrics for your integration.
To do so, follow these steps:
Get the integration object from the API:
curl --request GET https://api.<realm>.signalfx.com/v2/integration?type=AWSCloudWatch&offset=0&limit=50&orderBy=-lastUpdated
--header "X-SF-TOKEN:" \
--header "Content-Type:application/json" > integration.json
You’ll get something similar to:
{
"count": 2,
"results": [
{
"authMethod": "ExternalId",
"created": 1674862496869,
"createdByName": null,
"creator": "E-tkECKAsAA",
"customCloudWatchNamespaces": null,
"enableAwsUsage": true,
"enableCheckLargeVolume": true,
"enabled": false,
"externalId": "fyprhjmtpxttxwqhotep",
"id": "integration-id",
"importCloudWatch": true,
"largeVolume": false,
"lastUpdated": 1674862497253,
"lastUpdatedBy": "E-tkECKAsAA",
"lastUpdatedByName": "John Smith",
"name": "AWS Dev",
"pollRate": 300000,
"regions": [],
"roleArn": null,
"services": [],
"sfxAwsAccountArn": "arn:aws:iam::134183635603:root",
"syncCustomNamespacesOnly": false,
"syncLoadBalancerTargetGroupTags": false,
"type": "AWSCloudWatch"
},
{
"authMethod": "ExternalId",
"created": 1522297476849,
"createdByName": null,
"creator": "CGa4fY-AoAA",
"customCloudWatchNamespaces": null,
"enableAwsUsage": true,
"enableCheckLargeVolume": false,
"enabled": true,
"externalId": "uoejtvhsjnbcbdbfvbhg",
"id": "DZTsWRwAkAA",
"importCloudWatch": false,
"largeVolume": false,
"lastUpdated": 1671440367214,
"lastUpdatedBy": "CGa4fY-AoAA",
"lastUpdatedByName": "John Doe",
"name": "AWS Prod",
"pollRate": 300000,
"regions": [],
"roleArn": "arn:aws:iam::123456789012:role/splunk-o11y-role",
"services": [],
"sfxAwsAccountArn": "arn:aws:iam::134183635603:root",
"syncCustomNamespacesOnly": false,
"type": "AWSCloudWatch"
}
]
}
Modify the integration file as explained in steps 3 and 4.
Remove the fields below from the call, as these will be populated automatically:
``created``
``createdByName``
``creator``
``lastUpdated``
``lastUpdatedBy``
``lastUpdatedByName``
Include
ignoreAllStatusMetrics, set tofalse, in the integration. It will look like this:
{
"authMethod": "ExternalId",
"customCloudWatchNamespaces": null,
"enableAwsUsage": true,
"enableCheckLargeVolume": true,
"enabled": false,
"externalId": "fyprhjmtpxttxwqhotep",
"id": "integration-id",
"ignoreAllStatusMetrics": false,
"importCloudWatch": true,
"largeVolume": false,
"name": "AWS Dev",
"pollRate": 300000,
"regions": [],
"roleArn": null,
"services": [],
"sfxAwsAccountArn": "arn:aws:iam::134183635603:root",
"syncCustomNamespacesOnly": false,
"syncLoadBalancerTargetGroupTags": false,
"type": "AWSCloudWatch"
}
Update the integration object above using the API:
curl --request PUT https://api.<realm>.signalfx.com/v2/integration/<integration-id>
--header "X-SF-TOKEN:" \
--header "Content-Type:application/json" \
--data "@integration.json"
StatusCheckFailedis always ignored but now you can combine the other two status check metrics,StatusCheckFailed_InstanceandStatusCheckFailed_System, to obtain status information.For more on AWS status check metrics, see the official AWS documentation.