Collect logs from your AWS services π
When setting up an AWS connection, you can choose to import logs from a Cloudwatch log group or an S3 bucket.
To set up log collection, follow these steps:
Open the link to a CloudFormation template.
Adjust the settings.
Deploy the template to create
splunk-aws-logs-collector
, an AWS Lambda function used to transform log entries, enrich them with metadata, and send them to Splunk Observability Cloud.
How does log collection work? π
The Splunk Observability Cloud back end runs a periodic job which goes through CloudWatch log groups and services in your account. This job adds the appropriate subscriptions and notifications to trigger the splunk-aws-logs-collector
function.
Splunk Observability Cloud adds subscription filters to log groups for the selected services in the integration, or for all of the supported services when none is selected. For instance, if you select AWS/Lambda
in the integration, Observability Cloud will add subscription filters to /aws/lambda/*
log groups only. Splunk Observability Cloud doesnβt capture logs from all CloudWatch log groups.
Managing subscriptions π
Subscriptions are managed every 5 minutes, which is not configurable at the moment. If you decide to turn off the integration or a particular service, the job will attempt to remove those subscriptions.
If a new log group is created for a service in the integration, Observability Cloud will add a subscription filter to this newly created log group. Afterwards, whenever new log events are added to the log group, AWS triggers splunk-aws-logs-collector
lambda automatically in near real time.
Which services can you collect logs from? π
You can collect logs from the following services:
Services storing their logs in Cloudwatch. Logs are stored in log groups which start with
aws/<servicename>
. For example:aws/lambda
,aws/rds
, oraws/eks
WAF CloudWatch logs
API Gateway execution logs
AWS Glue continuous logs if a default log group name is used
Network and Application Load Balancers access logs (classic Load Balancers not supported) from S3
S3 access logs from S3
Redshift access logs from S3
CloudFront access logs from S3
Limitations π
The following restrictions apply:
Log sync can only be enabled for a single AWS integration per AWS account. Note the integration may cover multiple services and regions.
Deployment in China or Gov regions requires additional manual steps. See the available CloudFormation templates on GitHub.
Collect logs from unsupported services π
CloudWatch log groups also store logs from unsupported services. If you want to capture those logs, add /aws/<namespace>
to the list of custom namespaces in the integration object. While this option is not available in the Splunk Observability UI, you can easily do it via API, or by adding subscription filters.
Collect logs via API π
To capture logs from unsupported services via the API, follow these steps:
Use a
GET
request to retrieve existing integration object:
curl https://app.<realm>.signalfx.com/v2/integration/<integrationId> \
-H 'x-sf-token: <user API access token>'
Update the retrieved object by adding or modifying the
customNamespaceSyncRules
field by executing:
{
"customNamespaceSyncRules": [
{
"namespace": "aws/<namespace>"
}
],
"enabled": true,
"id": "E1c1_huAAAA",
}
Namespaces must use lowercase only
Some fields are omitted for brevity
Use a
PUT
request to update your integration:
curl https://app.<realm>.signalfx.com/v2/integration/<integrationId> \
-H PUT \
-H 'x-sf-token: <user API access token>' \
-H 'content-type: application/json' \
--data-raw '<updated integration JSON here>'
Collect logs manually with subscription filters π
Alternatively, you can add a subscription filter to selected CloudWatch log groups on your own. You may manually add the splunk-aws-logs-collector
lambda as a CW log group subscriber. You can use any name as a subscription filter name except for Splunk Log Collector, since such subscriptions are managed by Splunk Observability, and they would be removed automatically.
Metadata π
Log events from AWS services are enriched with relevant metadata. Some of the metadata is common to all services, while some other is service-specific.
Common metadata π
Field name |
Description |
Example |
---|---|---|
awsAccountId |
The AWS Account ID of the resource that produced the logs |
awsAccountId: 123456790 |
region |
The AWS region of the resource that produced the logs |
region: us-east-1 |
logForwarder |
The name and version of |
logForwarder: splunk_aws_log_forwarder:1.0.1 |
Service-specific metadata π
Services that store logs in CloudWatch Logs π
Field name |
Description |
Example |
---|---|---|
host |
Same as |
logGroup: /aws/lambda/my_function |
logGroup |
Source CloudWatch log group name |
logGroup: /aws/lambda/my_function |
logStream |
Source CloudWatch log stream name |
logStream: 2020/07/31/[1]e46fcdcac7094436bd846edb431a3f1 |
source |
Service name |
source: lambda |
sourcetype |
aws: prefixed service name |
sourcetype: aws:lambda |
API Gateway, ApplicationELB, CloudFront, EKS, Lambda, NetworkELB, RDS, Redshift, S3 π
Field name |
Description |
Example |
---|---|---|
[aws_tag_name] |
AWS tags associated with the resource that generated logs |
name: my_func_name env: prod myCustomTag: someValue |
API Gateway π
Field name |
Description |
Example |
---|---|---|
arn |
API gateway ARN |
arn: arn:aws:apigateway:us-east-1::/restapis/kgiqlx3nok/stages/prod |
host |
arn |
host: arn:aws:apigateway:us-east-1::/restapis/kgiqlx3nok/stages/prod |
apiGatewayStage |
The API Gateway Stage name |
apiGatewayStage: prod |
apiGatewayId |
The API Gateway ID |
apiGatewayId: kgiqlx3nok |
Application Load Balancer π
Field name |
Description |
Example |
---|---|---|
elbArn |
Load balancer ARN |
elbArn: arn:aws:elasticloadbalancing:us-east-1:1234567890:loadbalancer/app/my-loadbalancer/50dc6c495c0c9188 |
targetGroupArn |
Target group ARN (when available) |
targetGroupArn: arn:aws:elasticloadbalancing:us-east-1:1234567890:loadbalancer/app/my-loadbalancer/50dc6c495c0c9188 |
CloudFront π
Field name |
Description |
Example |
---|---|---|
distributionArn |
CloudFront distribution ARN |
distributionArn: arn:aws:cloudfront::1234567890:distribution/EMLARXS9EXAMPLE |
EKS π
Field name |
Description |
Example |
---|---|---|
arn |
EKS cluster ARN |
arn: arn:aws:rds:us-east-1:123456790:cluster/test-eks-cluster |
host |
EKS cluster host |
host: test-eks-cluster |
eksClusterName |
The EKS cluster name |
eksClusterName: test-eks-cluster |
Lambda π
Field name |
Description |
Example |
---|---|---|
arn |
The ARN of the lambda function that generated the logs |
arn: arn:aws:lambda:us-east-1:123456790:function:my_function |
host |
Lambda host |
host: arn:aws:lambda:us-east-1:123456790:function:my_function |
functionName |
The name of the lambda |
functionName: my_function |
Network Load Balancer π
Field name |
Description |
Example |
---|---|---|
elbArn |
Load balancer ARN |
elbArn: arn:aws:elasticloadbalancing:us-east-1:1234567890:loadbalancer/net/my-netlb/c6e77e28c25b2234 |
RDS PostgreSQL π
Field name |
Description |
Example |
---|---|---|
arn |
DB host ARN |
arn: arn:aws:rds:us-east-1:123456790:db:druid-lab0 |
host |
The host of RDS |
host: druid-lab0 |
dbType |
The type of DB |
dbType: postgresql |
RDS, other than PostgreSQL π
Field name |
Description |
Example |
---|---|---|
arn |
DB host ARN |
arn: arn:aws:rds:us-east-1:123456790:db:test-database-1 |
host |
The host of RDS |
host: test-database-1 |
dbLogName |
The name of the RDS log |
dbLogName: error |
Redshift π
Field name |
Description |
Example |
---|---|---|
clusterArn |
Redshift cluster ARN |
clusterArn: arn:aws:redshift:us-east-1:1234567890:cluster:redshift-cluster-1 |
logType |
Redshit log type. Possible: connectionlog, useractivitylog, or userlog |
logType: userlog |
S3 π
Field name |
Description |
Example |
---|---|---|
bucketArn |
S3 bucket ARN |
bucketArn: arn:aws:s3:::my-bucket |
objectArn |
S3 object ARN (when available) |
objectArn: arn:aws:s3:::my-bucket/sample.jpeg |
Troubleshooting π
CloudFormation stack was not created π
You fully control the process of creating the CloudFormation stack, which is executed with the permissions associated with your user. The template contains a lambda function and a role required to forward logs from Cloudwatch and S3 buckets. If any errors occur, AWS displays a specific error message.
To learn more about supported templates, see the README on GitHub.
I created an integration, but I donβt see any logs π
If you created the integration recently, it may take some time for the logs to appear in your account. The job that makes your logs notify Splunk AWS Log Collector runs every 5 minutes, so it might take that long to subscribe to a new resource. AWS logs delivery inside AWS (to CloudWatch log groups, or to S3 buckets) and AWS lambda triggering can introduce additional delay. Check AWS documentation for more details.
If you still donβt see any logs after 15 minutes, check the IAM policy youβve used to set up the AWS connection. We recommend using the provided IAM policy. If you still donβt see the logs, please contact our support.
You can enable debug mode on the log forwarding lambda function: Add LOG_LEVEL=DEBUG
in the Configuration > Environment variables section. If you see log forwarding calls fail due to a 503 HTTP error, you may be exceeding logs limit. To fix this, contact our support.
CloudFront access logs are not being collected π
CloudFront is a global service, and its logs can be stored in any of the standard AWS regions. Each CloudFront instance can have an S3 target bucket to access configured logs. Splunk AWS log collection can only grab the logs if the S3 bucket is located in a region Splunk AWS log collection can access.Use the provided IAM policy to ensure the Splunk Observability Cloud back end has the required permissions.
I donβt see logs from some instances π
Make sure your IAM policy allows access to the instances, their regions, or the regions where they send logs. If the service instance was recently created, it might take up to 15 minutes for the Splunk Observability Cloud back end to start gathering logs from it.
AWS allows you to configure only one notification of a given kind when a new log file appears, and S3 event files are created. If the bucket where an instanceβs logs are stored already notifies another lambda function of a file creation, Observability Cloud cannot add its subscription on top of that. You can either remove the pre-existing notification configuration, or narrow it by specifying a prefix and a suffix in such a way that the log files wonβt be triggering your pre-existing lambda function. If thatβs not possible, contact us for assistance to modify your AWS architecture to work around the limitation.
I donβt see logs from some of my S3 buckets π
Some AWS services use S3 buckets to store their logs, and sometimes the S3 bucket is located in a different region from the service that produces those logs. In such cases make sure to deploy the splunk-aws-logs-collector
lambda function using the CloudFormation template in all AWS regions where S3 buckets with logs are located.
I have disabled logs collection, but logs are still gathered by Observability Cloud π
It may take up to 15 minutes for the Observability Cloud back end to cancel log subscriptions. There may be additional delays introduced by the AWS logs delivery process.
The back end needs log related permissions to cancel log subscriptions. If log related permissions are removed from the AWS IAM policy (or the entire policy is removed), the back end cannot run the cleanup procedure. Make sure to disable the log collection on Observability Cloudβs side first, and clean up on AWSβ side later.
I disabled the integration or changed its settings, but logs are still being collected! π
If you disable a part or all the integration, our back end job will attempt to clear all notifications and subscriptions it has previously created, which might take up to 15 minutes. However, if you also remove IAM permissions, the attempt may fail.
To stop sending any logs to Observability Cloud, delete the Splunk AWS Logs collector lambda from the region where you wish to stop collecting logs.