Splunk Enterprise does not start due to unusable filesystem

If you receive an error message like the following when you start Splunk Enterprise on a *nix machine, it might be because the software does not know how to write to your machine filesystem.

homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem. Validating databases (splunkd validatedb) failed with code '1'.

Splunk Enterprise must be able to write to the local filesystem to index your data. Splunk provides support for many different filesystems, as described in System requirements in the Installation Manual. On machines with an unrecognized filesystem, Splunk Enterprise runs a utility called locktest that confirms whether it can work with the filesystem. If locktest fails for any reason, splunkd does not start, to prevent you from indexing data to a filesystem that it cannot write to.

The locktest utility can fail for a number of reasons:

• The filesystem is not known, and Splunk Enterprise cannot perform the proper file locking on it.
• The filesystem has been marked as read-only, or has otherwise been changed by the operating system.
• A library or function that locktest uses to perform the tests is not available or cannot be loaded.

This troubleshooting topic does not apply to Splunk Enterprise instances that run on Windows machines.

Temporarily bypass filesystem checks

If you are a Splunk administrator who understands the risks, you can temporarily bypass filesystem checks to get Splunk Enterprise running again.

Configuring this setting can be dangerous and is not supported in normal operations. Irrevocable data loss can occur. You perform this action solely at your own risk. By configuring the setting, you actively bypass filesystem checks that confirm if Splunk Enterprise can run on your machine filesystem. In a production environment, you must not use this setting as a long-term solution to a filesystem problem. If you use the setting under the guidance of Splunk Support, immediately report any problems that you encounter with indexing or search.

Use the setting in one or more of the following scenarios only:

• You are a skilled Splunk administrator and understand the risks of bypassing filesystem checks.
• You use Splunk software in a development environment.
• You want to recover from a situation where the default filesystem has been changed outside of your control, such as during an operating system upgrade.
• You want to recover from a situation where a Splunk bug has invalidated a previously functional filesystem after an upgrade.
• You want to evaluate the performance of a filesystem for which Splunk has not yet offered support.
• You have been given explicit instruction from Splunk Support to use the setting to solve a problem where Splunk software does not start because of a failed filesystem check.
• You understand and accept all of the risks of using the setting, up to and including losing all your data with no ability to recover it.

1. On the machine that is experiencing the failure, open a shell prompt.
2. Become root or an administrative equivalent with su:
   

   sudo su -
   

3. Open $SPLUNK_HOME/etc/splunk-launch.conf with a text editor.
   

   $SPLUNK_HOME represents where you have installed Splunk Enterprise. For example, if you installed Splunk Enterprise in /opt/splunk, then you would edit /opt/splunk/etc/splunk-launch.conf.

4. In the file, add the following line anywhere:
   

   OPTIMISTIC_ABOUT_FILE_LOCKING=1
   

5. Save the file and close the text editor.
6. Restart Splunk Enterprise.
7. Confirm that the splunkd service has started.