Splunk® Enterprise

Managing Indexers and Clusters of Indexers

Acrobat logo Download manual as PDF


Splunk Enterprise version 7.1 is no longer supported as of October 31, 2020. See the Splunk Software Support Policy for details. For information about upgrading to a supported version, see How to upgrade Splunk Enterprise.
Acrobat logo Download topic as PDF

Non-clustered bucket issues

This section tells you how to deal with an assortment of bucket problems that can exist independent of clustering.

Rebuild all buckets

The indexer usually handles crash recovery without your intervention. If an indexer goes down unexpectedly, some recently received data might not be searchable. When you restart the indexer, it will automatically run the fsck command in the background. This command diagnoses the health of your buckets and rebuilds search data as necessary.

Caution: It is unlikely that you will need to run fsck manually. This is a good thing, because to run it manually, you must stop the indexer, and the command can take several hours to complete if your indexes are large. During that time your data will be inaccessible. However, if Splunk Support directs you to run it, the rest of this section tells you how to do so.

To run fsck manually, you must first stop the indexer. Then run fsck against the affected buckets. To run fsck against buckets in all indexes, use this command:

splunk fsck repair --all-buckets-all-indexes

This will rebuild all types of buckets (hot/warm/cold) in all indexes.

To rebuild all buckets in just a single index, use this version of the command:

splunk fsck repair --all-buckets-one-index --index-name=<your_index>

Note: The fsck command only rebuilds buckets created by version 4.2 or later of Splunk Enterprise.

The fsck repair command can take several hours to run, depending on the size of your indexes If you determine that you only need to rebuild a few buckets, you can run the rebuild command on just those buckets, as described in the next section, Rebuild a single bucket.

If you just want to diagnose the state of your indexes (without taking any immediate remedial action), run:

splunk fsck scan --all-buckets-all-indexes

To learn more about the fsck command, including a list of all options available, enter:

splunk fsck --help

Rebuild a single bucket

If the index and metadata files in a bucket (version 4.2 and later) somehow get corrupted, you can rebuild the bucket from the raw data file alone. Use this command:

splunk rebuild <bucket directory> <index-name>

The indexer automatically deletes the old index and metadata files and rebuilds them. You don't need to delete any files yourself.

Note:

  • Rebuilding a bucket does not count against your license.
  • The time required to rebuild a bucket can be significant. Depending on various system considerations, such as your hardware specifications, it can take anywhere from half an hour to a few hours to rebuild a 10 GB bucket.
  • splunk rebuild is an alias of splunk fsck repair --one-bucket.

Recover invalid pre-4.2 hot buckets

A hot bucket becomes an invalid hot (invalid_hot_<ID>) bucket when the indexer detects that the metadata files (Sources.data, Hosts.data, SourceTypes.data) are corrupt or incorrect. Incorrect data usually signifies incorrect time ranges; it can also mean that event counts are incorrect.

The indexer ignores invalid hot buckets. Data does not get added to such buckets, and they cannot be searched. Invalid buckets also do not count when determining bucket limit values such as maxTotalDataSizeMB. This means that invalid buckets do not negatively affect the flow of data through the system, but it also means that they can result in disk storage that exceeds the configured maximum value.

To recover an invalid hot bucket, use the recover-metadata command:

1. Make backup copies of the metadata files, Sources.data, Hosts.data, SourceTypes.data.

2. Rebuild the metadata from the raw data information:

splunk cmd recover-metadata path_to_your_hot_buckets/invalid_hot_<ID>

3. If successful, rename the bucket as it would normally be named.

Rebuild index-level bucket manifests

The index-level bucket manifest file is .bucketManifest. It contains a list of all buckets in the index.

It is unusual to need to rebuild the manifest. One situation where you might need to do so is if you manually copy a bucket into an index.

Only rebuild the manifest if Splunk Support directs you to. Do not rebuild it on your own.

This command rebuilds the .bucketManifest file for the main index only:

splunk _internal call /data/indexes/main/rebuild-bucket-manifest

To rebuild the manifests for all indexes, use the asterisk (*) wildcard:

splunk _internal call /data/indexes/*/rebuild-bucket-manifest
Last modified on 07 July, 2022
PREVIOUS
What happens when the master node goes down
  NEXT
Bucket replication issues

This documentation applies to the following versions of Splunk® Enterprise: 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 7.3.6, 7.3.7, 7.3.8, 7.3.9, 8.0.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.0.5, 8.0.6, 8.0.7, 8.0.8, 8.0.9, 8.0.10, 8.1.0, 8.1.1, 8.1.2, 8.1.3, 8.1.4, 8.1.5, 8.1.6, 8.1.7, 8.1.8, 8.1.9, 8.1.10, 8.1.11, 8.1.12, 8.1.13, 8.1.14, 8.2.0, 8.2.1, 8.2.2, 8.2.3, 8.2.4, 8.2.5, 8.2.6, 8.2.7, 8.2.8, 8.2.9, 8.2.10, 8.2.11, 8.2.12, 9.0.0, 9.0.1, 9.0.2, 9.0.3, 9.0.4, 9.0.5, 9.0.6, 9.0.7, 9.0.8, 9.1.0, 9.1.1, 9.1.2, 9.1.3, 9.2.0


Was this documentation topic helpful?


You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters