Docs » Manage the logs pipeline » Infinite Logging rules

Infinite Logging rules 🔗

Create Infinite Logging rules to archive all or any subset of logs in Amazon S3 buckets for compliance or possible future use while not paying to index them unless and until you want to analyze them in Splunk Log Observer. Storing all logs in S3 buckets with a retention time that you control helps you meet compliance and audit requirements.

Some logs may not be useful on a day-to-day basis but may still be important in case of a future incident. For example, you might want to exclude logs from a non-production environment or debug logs when indexing. In either case, you can create an Infinite Logging rule to archive those logs in S3 buckets that your team owns in AWS.

Prerequisites 🔗

You must be a Splunk Observability Cloud admin to create new Infinite Logging connections. Non-admins can send data to S3 buckets using an existing Infinite Logging connection, but they cannot create new connections. See AWS documentation for permissions required to create S3 buckets in the AWS Management Console.

Create an Infinite Logging rule 🔗

To create an Infinite Logging rule, follow these steps:

  1. From the navigation menu, go to Organization Settings > Logs Pipeline Management.

  2. Click New Infinite Logging Rule.

  3. Decide where your logs should go on the Define Forwarding Behavior page, then click Next.

    Select Archive and index to archive your logs in an Amazon S3 bucket and also index them in Observability Cloud where you can analyze them in Splunk Log Observer. Use this option if, for example, you want to keep all logs from env:production in Log Observer for troubleshooting, but you also want to retain them in an archive for longer term storage. The number of logs you index impacts billing.

    Select Archive without indexing to archive your logs in an S3 bucket only. Use this option when, for example, you don’t want to index any of the env:dev/test logs. If you don’t want to index any logs, then do not set up a sampling rate on the Filter Data page. This will send the logs data to the archive. Note: When you do not index logs, you cannot analyze logs or the patterns they create in Log Observer. Archiving logs without indexing them in Observability Cloud does not impact your billing.

  4. To send your data to an existing S3 bucket, click the Infinite Logging connection you want, then skip to step 10.

  5. If you want to send your data to a new S3 bucket and you are an Observability Cloud admin, click Create new connection. The Establish a New S3 Connection wizard appears.

  6. On the Choose an AWS Region and Authentication Type tab, do the following:

    1. Select the AWS region you want to connect to.

    2. Select whether you want to use the External ID or Security Token authentication type.

    3. Click Next.

  7. On the Prepare AWS Account tab, follow the steps in the wizard to do the following in the AWS Management Console:

    1. Create an AWS policy. The wizard provides the exact policy you must copy and paste into AWS.

    2. Create a role and associate it with the AWS policy.

    3. Create and configure an S3 bucket.

  8. On the Establish Connection tab, do the following:

    1. Give your new S3 connection a name.

    2. Paste the Role ARN from the AWS Management Console into the Role ARN field in the wizard.

    3. Give your S3 bucket a name.

    4. Click Save.

  9. Choose the Amazon S3 Infinite Logging connection that you created in step 1 of the wizard. Your data will go to your S3 bucket in a file that you configure in the following two steps.

  10. (Optional) You can add a file prefix, which will be prepended to the front of the file you send to your S3 bucket.

  11. (Optional) In Advanced Configuration Options, you can select the compression and file formats of the file you will send to your S3 bucket.

  12. Click Next.

  13. On the Filter Data page, create a filter that matches the log lines you want to archive in your S3 bucket. Click (Optional) Define a sampling percentage if you want to index a percentage of the logs that will be sent to S3. Indexing a small percentage of logs allows you to see trends in logs that are being archived in S3 buckets. Adjust the Sampling Percentage using the slider.

Logs returned by this filter will follow the forwarding behavior you defined in step 3. If you don’t want to index any logs, then do not set up a sampling rate. Click Next when you are finished.

  1. Add a name and description for your Infinite Logging rule.

  2. Review your configuration choices, then click Save.

Your Infinite Logging setup is now complete. Depending on your selections, your logs will be archived, indexed in Observability Cloud for analysis, or both.