Splunk Cloud

Splunk Cloud User Manual

Download manual as PDF

Download topic as PDF

Archive expired Splunk Cloud data

You might need to maintain older data to access it or for compliance purposes. Dynamic Data Active Archive allows you to move your data from your Splunk Cloud indexes to a Splunk-maintained archive. You specify archiving at the index level; i.e., you create an archiving rule for a specified index. This allows you the flexibility to only archive the data that you need to archive. You can configure Splunk Cloud to automatically move the data from an index when the data reaches the end of the Splunk Cloud retention period you configure. In addition, you can restore your data to your Splunk Cloud environment for searching.


Browser Requirements for Dynamic Data Active Archive

Dynamic Data Active Archive supports the same browsers and browser versions supported by Splunk Cloud:

  • Firefox ESR (24.2 and current version)
  • Internet Explorer 9, 10, and 11 (compatibility mode is not supported)
  • Safari (current version)
  • Chrome (current version)

How Dynamic Data Active Archive works

Data is moved to the archive when the index meets a configured size or time threshold. You may have configured the data to be stored for a certain number of days or until the index reaches a certain size. When that threshold is met, Splunk Cloud attempts to move the data to the archive location. If an error occurs, if there are connection issues, Splunk Cloud attempts to move the data every 15 minutes until it can successfully move it.

It can take up to 48 hours from the archive initiation for the archiving process to complete.

If an error occurs, the error is logged to the splunkd.log. Splunk Cloud does not delete data from the Splunk Cloud environment until is has successfully moved the data to the archive. If you need to restore the data so that it is searchable, you can restore the data to your Splunk Cloud environment. You can then search the data and delete it when you have finished.

When you restore archived data to Splunk Cloud, it does not count against the indexing license volume for the Splunk Cloud deployment.

Dynamic Data Active Archive Performance

When you restore data, it can sometimes impact performance if there is a large amount of data. Splunk Cloud has checks in place to help you determine whether the size of the data is too large to restore, and will provide a warning if the data size may impact performance. Splunk Cloud will block you from restoring an amount of data that could potentially have a very negative impact on performance. If this occurs, select a smaller time range.

Configure archive settings for an index

Configure archive settings for a specific index.

Managing archive settings requires the indexes_edit capability. All archiving changes are logged in the audit.log file.

Configure archiving for an index

  1. In Splunk Cloud, go to Settings > Indexes.
  2. Click New Index to create a new index or click Edit in the Actions column for an existing index.
  3. In the Dynamic Data Storage field, select Splunk Archive.
  4. Set the archive retention period. You can specify this value in years, months, or days. Note that the maximum archive retention period is displayed. This value is based on your licensed archive retention period. Specify a value within this range.
  5. Click Save.

Disable archiving for an index

  1. Go to Settings > Indexes.
  2. Click Edit in the Actions column for the index you want to manage.
  3. In the Dynamic Data Storage field, select Self Storage to move data to self-storage location when it expires or No Additional Storage to delete data as it expires.
  4. Click Save. When data in this index expires, it is deleted.

Disabling archiving for an index does not delete existing archived data. Existing archived data will be maintained until the configured expiration period. Disabling archiving for an index also does not affect the time or size of the data retention policy for the index.

Restore archived data to Splunk Cloud

You might need to restore indexed data from the Splunk archive. When the data is restored, you can then search it like any other data. You restore data based on the time period for the data you want to search. For example, you might want to restore data for a period of a day. When you pick a date from the date-picker, it is treated as 12 a.m. UTC of the selected date. So, if you want to restore one day's worth of archived data, (for example, on 7/10/2018) you would need to specify 07/10/2018 in the 'from' field and 07/11/2018 in the 'to' field. By default, restored data is searchable for a period of a month and is removed from Splunk after this period. It is not removed from the archive.

After you initiate data restoration, it can take up to 24 hours before data is restored. If it takes longer than 24 hours, contact Splunk Technical Support.

How restoring data works

When you restore data to Splunk Cloud from the archive, a copy of the archived data is moved back to the Splunk Cloud environment. To ensure your data is safe, the original archived data is never moved or deleted. This method of temporary data restoration ensures that you can never mistakenly delete your archived data.

When you restore archived data to an index in your Splunk Cloud instance, it does not count against the retention periods configured for data in your index. Restored data exists outside of the constraints of retention periods and size limits and does not affect the retention of your existing index data.

When you restore data, Splunk Cloud checks several conditions to ensure that you do not experience performance issues and that you do not duplicate data and cause your queries to return incorect results:

  • Check for overlapping data. Splunk Cloud does not restore data if you have already restored data in that same time range. This is to ensure you do not restore duplicate data, which would cause inaccurate search results. For example, if you specify that you want to restore data from 7/1/2018 - 7/3/2018 but you have already restored data from 7/1/2018 - 7/2/2018, Splunk Cloud will prevent your data restore. In this case, it is recommended that you restore that data that falls outside of the range of the data you have already restored. In this example, you would restore data from 7/3/2018-7/4/2018.
  • Check to ensure data is not likely to cause performance issues. Splunk Cloud checks the size of the data you want to restore and presents you with a warning if the size of the data may cause performance issues. If the size of that data is very likely to cause performance issues, Splunk Cloud will prevent you from restoring the data.

Because there is a time period during which data is being transitioned from Splunk Cloud to the archive, you will not be able to restore that data during the processing period. Generally, the data moved to the archive is available in approximately 48 hours. If your attempt to restore archived data fails, verify that the data was not recently archived.

After you have restored data, you may notice that events appear in your index that are older than your configured retention period specifies. This restored data will remain in your index for 30 days or until you clear it.

What happens when you are finished searching the restored data

After the data is temporarily restored to your Splunk Cloud environment it is available for searching for 30 days. Restored data is a copy of the archived data so you never need to move the data back to the archive, but for best performance, you should remove the temporarily restored data when you have finished searching it.

Steps to restore data to Splunk Cloud

  1. In Splunk Cloud, go to Settings > Indexes.
  2. For the index where you want to restore data, click Restore. The menu displays the restore history for the specified index. You can see the history of data restoration and file size for the data restored.
  3. Use the date picker to select a time range to retrieve.
  4. Click Check size. Splunk Cloud checks to see if the size of the file might impact performance. If the file size is too large, Splunk Cloud blocks you from restoring data. If there is a potential performance impact, Splunk Cloud displays a warning. Splunk Cloud also prevents you from restoring data that overlaps with existing restored data.
  5. Enter an email address to send job status notifications. Because it can take up to 24 hours to restore data, enter your email address to enable Splunk Cloud to notify you when restoration is complete.
  6. Click Restore when you have refined the file size or date range to acceptable limits.

    It can take up to 24 hours for data to be restored.

  7. To check the status of your data restoration, click Splunk Archive in the Storage Type field to open the Archive page. To view the restore status, click the Restore tab. In the JobStatus field, you can see the status of your job: Pending (means that the job has been submitted, but has not begun processing), In progress (means that the job has been started, and is progressing), Success, Cleared (means you successfully deleted the temporary archive from your index), Expired (means that the restored data has passed the 30 day retention period and has been deleted from the index), and Failed. If you receive a Failed status, click the > button for the archive to display more details about why the restoration failed.

Steps to remove restored data from Splunk Cloud

Splunk recommends you manually remove restored data when you are finished searching it.

Restored data is a copy of the archived data, so you never need to move the data back to the archive, but for best performance, you should remove the temporarily restored data when you are done searching it.

To remove restored data:

  1. In Splunk Cloud, go to Settings > Indexes.
  2. Select the index with data you want to remove and click Restore to open the Restore Archive page.
  3. For the range of data you want to remove, select Clear in the Actions column.

When the data is successfully removed, the Jobstatus column displays a Cleared status.

Monitor Dynamic Data Active Archive

Splunk generates logs when you archive data and when you restore archived data. You may want to monitor these logs to check for errors during these processes.

Archiving logs

To check for error messages that occur when you are archiving data, you can view the coldstoragearchiver entries in the splunkd.log. You can find these entries by running the following search:

index=_internal source=*/splunkd.log component=coldstoragearchiver

Data restoration logs

To check for error messages that occur when you restore archived data, you can view entries in the splunk_archiver_restoration.log, restoration.log, and python.log. You can find these entries by running the following search:

index=_internal source=*/splunk_archiver_restoration.log

index=_internal source=*/restoration.log

index=_internal source=*/python.log

Manage your archives

You can review the status of your archived indexes on the Archived Indexes page.

Steps to review your archived indexes

  1. In Splunk Cloud, go to Settings > Indexes.
  2. Click Archived Indexes for an archived index to open the Archived Indexes page.
  3. Review the indexes you have archived on the Archived tab.
  4. To see the restore history for your indexes, click the Restore tab.

On the Restore tab, you can see the indexes that you have restored and their status:

  • Pending. The request for restoration has been initiated, but has not yet begun.
  • In progress. The restoration process has started, but it has not been completed.
  • Success. The data has been successfully restored to your index.
  • Failure. The restoration failed. Click the > button next to the archive to display more details about the failure.
  • Cleared. You have successfully cleared the temporarily restored data.
  • Expired. The restored data has passed the 30 day retention threshold

After you have reviewed the archived indexes, you can determine what actions you want to take for each archived or restored index. You may want to clear archived data or stop archiving an index. Or you may see that a restoration or archive operation failed and chose to troubleshoot the issue.

Troubleshoot Dynamic Data Active Archive

I received an error when attempting to restore data

If an error occurs, the error is logged to the splunkd.log. When you review the Archive page, if you experience errors, you may want to review the splunkd.log and specify the coldstoragearchiver component here: index=_internal source=*/splunkd.log component=coldstoragearchiver

I clicked the Check Size button and nothing happened

When restoring data, I clicked the Check Size button multiple times and nothing happened.

Diagnosis

When restoring a large amount of data, it may take some time for Splunk to verify that the size of the data can be restored without causing performance issues. If you click the Check Size buttons multiple times, it may trigger AWS to block the check process.

Solution

Do not click the Check Size button multiple times if you don't immediately receive feedback.

I archived some of my data. When I attempted to restore it a few hours later, the restoration failed.

When I archived data, I attempted to restore it soon after, and the restoration failed.

Diagnosis

Data can take up to 48 hours to archive. If you attempt to restore the data before this time period completes, the restoration will fail.

Solution

Wait until the 48 hour threshold has been met, and then attempt to restore the data.

PREVIOUS
Store expired Splunk Cloud data
  NEXT
Manage Splunk Cloud users and roles

This documentation applies to the following versions of Splunk Cloud: 7.2.3, 7.2.4, 7.2.6, 7.2.7


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters