Configure the S3 remote store for SmartStore

Before you configure SmartStore settings on the indexers, you must ensure that your remote store is properly set up, so that it is available to the indexers.

Later, when you configure remote volumes for SmartStore, you configure settings specific to the remote store in indexes.conf. The indexer uses those settings to communicate with the remote store.

Supported remote storage services

Supported remote storage services include S3, Google GCS, and Microsoft Azure Blob storage. For information on GCS, see Configure the GCS remote store for SmartStore. For information on Azure Blob storage, see Configure the Azure Blob remote store for SmartStore

For S3, SmartStore can use:

• AWS S3
• S3-API-compliant object stores

To determine whether your object store is S3-compliant, use the S3 compatibility checking tool, located here: https://github.com/splunk/s3-tests. To use the tool, follow the instructions in the repository's README file.

Configure an S3 remote store

When configuring S3 buckets:

• The buckets must have read, write, and delete permissions.
• If the indexers are running on EC2, provision the buckets for the same region as the EC2 instances that use it.
• The S3 buckets must be used only by SmartStore. Do not share S3 buckets with other tools such as ingest actions and edge processors.

See the Amazon S3 documentation for information on on how to create and configure buckets.

For S3-specific settings available through Splunk Enterprise, search for settings in the indexes.conf spec file that start with remote.s3.

For information on security-related settings, such as settings for S3 authentication and encryption, see SmartStore on S3 security strategies.

Remote store versioning support

Remote store versioning support is optional and turned on for SmartStore by default. Some third party remote stores do not support versioning and so the SmartStore capability must be turned off. Check with your third party vendor and turn off versioning if necessary.

To turn off versioning support, use the remote.s3.supports_versioning setting in indexes.conf.

Accommodate the remote store addressing model

Amazon S3 currently supports two addressing models, or request URI styles: path style, or V1, and virtual-hosted style, or V2. In V1, the bucket name is in the URI path; for example, //s3.amazonaws.com/<bucketname>/key. In V2, the bucket name is part of the domain name; for example, //<bucketname>.s3.amazonaws.com/key.

Amazon is deprecating support for V1 and, going forward, will require requests for new S3 buckets to use the V2 model.

Splunk Enterprise remote store addressing for native S3

Splunk Enterprise accommodates both V1 and V2 models automatically for Amazon S3 buckets.You can use either model, but Splunk Enterprise will convert V1 URIs to V2 when communicating with S3.

Using the V1 model, you specify the URI like this:

[volume:s3volume]
storageType = remote
path = s3://<bucketname>/rest/of/path

Using the V2 model, you specify the URI like this:

[volume:s3volume]
storageType = remote
path = s3://rest/of/path
remote.s3.bucket_name = <bucketname>

Similarly, if you specify an endpoint that ends in amazonaws.com, Splunk Enterprise determines the URI version from the endpoint, since the structure is fixed. For example:

[volume:s3volume]
storageType = remote
path = s3://<bucketname>/rest/of/path
remote.s3.endpoint = https://s3.us-west-1.amazonaws.com

or

[volume:s3volume]
storageType = remote
path = s3://rest/of/path
remote.s3.endpoint = https://<bucketname>.s3.us-west-1.amazonaws.com

Both of these specify the same bucket, and Splunk Enterprise will correctly resolve either one.

Splunk Enterprise remote store addressing for S3-compatible remote stores

If you use an S3-compatible remote store, rather than native S3, you might need to specify the addressing model that the S3-compatible store supports. You need to specify the model only if the S3-compatible remote store does not support V1.

Splunk Enterprise provides the remote.s3.url_version setting to specify the model that you are using. Its default value is v1. To change the addressing model to V2, change the setting to v2. For example:

[volume:s3volume]
storageType = remote
path = s3://rest/of/path
remote.s3.url_version = v2
remote.s3.endpoint = https://bucketname.whatever.customer.com