Archive Splunk indexes with the user interface
Before you begin, note the following:
- Splunk must be installed using the same user for all indexers and the Hunk Splunk Enterprise instances. This is the user which connects to HDFS for archiving and the user and user permissions must be consistent.
- The data in the referring Splunk Index must be in a warm or cold bucket in order to be archived.
- The Hunk Search Head and each Splunk indexer must have Hadoop client libraries and Java libraries in the same location. See System and software requirements in this manual for updated information about the required versions.
- The Splunk user associated with the Splunk indexer must have permission to write to the HDFS node.
- Hunk cannot currently archive buckets with raw data larger than 5GB to S3. You can configure your Splunk Enterprise bucket sizes in
indexes.conf. See Archiving Splunk indexes to S3 in this manual for known issues when archiving to S3.
Configure Splunk index archiving with the user interface
1. Navigate to Settings > Virtual Indexes and select the Archived Indexes tab. You can edit any existing archived index by clicking the arrow to its left.
2. Click New Archived Indexes to archive another Splunk Enterprise index into Hunk.
3. Type the names of the Splunk Enterprise indexes you want to archive in Hunk. You can add mulitple indexes. Splunk indexes that are already archived are disabled in the drop down list.
4. Provide a suffix for the new archive indexes. For example, if you select the "_archive" suffix, the new archived index will be "indexname_archive".
5. Select the Provider that the new archived indexes will be assigned to.
Note you can determine the bandwidth by provider that these archives can use. See "Set bandwidth limits for Hunk archiving" in this topic.
6. For Destination path in HDFS, provide the path to the working directory your provider should use for this data. For example: /user/root/archive/splunk_index_archive. If you are copying data to S3, prefix this path with:
7. Determine how old the data in the Splunk Enterprise index should be when it is copied to the archived index in Hunk. For example, if you select "5 Days," data is copied from the cold or warm bucket in the Splunk indexer to the Hunk archive bucket when it is five days old. Note: Splunk deletes data after a period of time defined in your indexer settings, so make sure that this field is set to copy the buckets before they are deleted by Splunk.
Set bandwidth limits for Hunk archiving
If you have concerns about the bandwidth required for consistent archiving, you can set bandwidth throttling. You set throttling for a provider, the limit you set for your provider is then applied across all indexes assigned to that provider.
Note: We currently cannot guarantee bandwitdh limits for bucket archival to S3 file systems.
To set bandwidth throttling:
1. In the Archived Indexes tab, click on Max bandwidth (Provider) for the index you want to edit. This opens the Edit Provider page for that index.
2. Under "Archive Settings" check Enable Archive Bandwidth Throttling.
3. Enter the maximum bandwidth you want to allow for all archived indexes associated with the provider.
4. Click Save.
Archive Splunk indexes using the configuration files
Archiving Splunk Enterprise indexes to Amazon S3
This documentation applies to the following versions of Hunk®(Legacy): 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11