Check the integrity of your Splunk software files

Most files that Splunk software ships with should not be modified by end users or administrators. However, many users mistakenly modify these files. For example, someone might edit a configuration file in the default directory, or files might be corrupted by hardware flaws, file system problems, a mangled installation, or an errant script.

File validation can identify when the contents of the files of a Splunk software instance have been modified in a way that is not valid. You can run this check manually, and it also runs automatically on startup. If you are an admin, you can view the results in a Monitoring Console health check or in a dashboard from any node.

Run the check manually

You might want to run the integrity check manually under any of the following conditions:

• You have problems after an upgrade.
• You have symptoms that make you suspect that there may have been a storage system problem.
• You suspect or wish to guard against the common error of edits to the default .conf files.
• As part of a regular system check. See Customize the health check in the Monitoring Splunk Enterprise manual.

To run the check manually with default settings, from the installation directory, type ./splunk validate files. You can manually run the integrity check with two controls.

• You can specify the file describing the correct file contents with -manifest. You might want to do this to check against an old manifest from a prior installation after a botched upgrade, to validate that the files are simply stale. You can use any valid manifest file. A manifest file ships in the installation directory with a new Splunk Enterprise download.
• You can constrain the test to only files that end with .conf by using -type conf. This is the set of messages the startup-time check prints to the terminal.

Options for automatic verification

The check runs at startup in two parts.

First, as part of the pre-flight check before splunkd starts, the check quickly validates only the default conf files and writes a message to your terminal.

Next, after splunkd starts, the check validates all files shipped with Splunk Enterprise (default conf files, libraries, binaries, data files, and so on). This more complete check writes the results to splunkd.log as well as to the bulletin message system in Splunk Web. You can configure it in limits.conf.

Options for the second part of the check in limits.conf include the following:

• run and log
• run, log, and emit a message to Splunk Web
• disable it

See limits.conf.spec.

Reading all the files provided with the installation has a moderate effect on I/O performance. If you need to restart Splunk software several times in a row, you might wish to disable this check temporarily to improve I/O performance.

Files are validated against the manifest file in the installation directory. If this file is removed or altered, the check cannot work correctly.

View results in Splunk Web

If you are an admin, you can view the results in a Monitoring Console health check or in a dashboard from any node. See Access and customize health check for more information about the Monitoring Console health check.

To view the default dashboard from any node:

1. Log in as admin to Splunk Web on any node in your deployment.
2. From Splunk Home, click Search & Reporting to enter the Search & Reporting app.
3. In the Apps bar, click Dashboards.
4. In the list of dashboards, click Integrity Check of Installed Files.

Interpret results of an integrity check

If an integrity check returns an error, such as "File Integrity checks found files that did not match the system-provided manifest", here are some tips to get you started resolving the problem.

• If the integrity check complains about conf files in default directories, determine how these files became changed and avoid this practice in the future. Modified default conf files will be overwritten on upgrade, creating hard-to-identify problems. See How to edit a configuration file for more details on how to edit configuration files in Splunk software.
• If it complains about files in $SPLUNK_HOME/bin or $SPLUNK_HOME/lib, or on Windows %SPLUNK_HOME%\Python2.7\, you probably need to reinstall. First try to find out how Splunk software was installed locally and determine whether this process could have resulted in a mix of files from different versions. AIX can cause this problem by holding library files open even after the Splunk service has been shut down. On most platforms this type of problem can occur when a Splunk product is upgraded while it is still running. If you cannot determine how this situation occurred, or how to resolve it, work with Splunk Support to identify the issue.
• If it cannot read some files, Splunk software may have been run as two or more different users or security contexts. Files created at install time under one user or context might not be readable by the service now running as another context. Alternatively, you might have legitimately modified the access rules to these files, but this is far less common.
• If the integrity check reports that it cannot read or comprehend the manifest, the manifest might be simply missing from $SPLUNK_HOME, or you have access problems to it, or the file may be corrupted. You might want to evaluate whether all the files from the installation package made it to the installation directory, and that the manifest contents are the same as the ones from the package. The manifest is not required for Splunk software to function, but the integrity check cannot function without it.
• If the integrity check reports all or nearly all files are incorrect, splunkd and etc/splunk.version might be in disagreement with the rest of the installation. Try to determine how this could have happened. It might be that the majority of the files are the ones you intended to be present.
• If the pattern is not described above, you might need to apply local analysis and troubleshooting skills possibly in concert with Splunk Support.

Interaction with monitoring console health check

The monitoring console health check queries the server/status/installed-file-integrity endpoint. This endpoint is populated with results when the integrity check runs at startup. See server/status/installed-file-integrity in the REST API Reference Manual.

If Splunk Enterprise starts with the integrity check disabled in limits.conf, then REST file integrity information is not available. In addition, manual runs do not update the results.

See Access and customize health check in Monitoring Splunk Enterprise.