About upgrading to 6.3 - READ THIS FIRST
This topic contains important information and tips about upgrading to version 6.3 from an earlier version. Read it before attempting to upgrade your Splunk environment.
Splunk App and Add-on Compatibility
Not all Splunk apps and add-ons are compatible with Splunk Enterprise 6.3. If you plan to upgrade to this release, visit Splunkbase to confirm that your apps are compatible with Splunk Enterprise 6.3.
Upgrade clustered environments
To upgrade an indexer cluster, read "Upgrade an indexer cluster" in the Managing Indexers and Clusters manual. The instructions in that topic supersede the upgrade material in this manual.
To upgrade a search head cluster, read "Upgrade a search head cluster" in the Distributed Search manual. The instructions in that topic supersede the upgrade material in this manual.
Splunk Enterprise supports the following upgrade paths to Version 6.3 of the software:
- From version 6.0 or later to 6.3 on full Splunk Enterprise.
- From version 5.0 or later to 6.3 on Splunk universal forwarders.
If you run a version of Splunk Enterprise prior to 6.0, upgrade to 6.0 first, then upgrade to 6.3. Users of Splunk Enterprise 5.0 also have the option of upgrading to versions 6.0, 6.1, or 6.2 before upgrading to 6.3. Read "About upgrading to 6.0 - READ THIS FIRST" for tips on migrating your instance to version 6.0.
Important upgrade information and changes
Here are some things that you should be aware of when installing the new version:
The working directory for the inputcsv, outputcsv, and streamedcsv search commands has changed
The working directory for the
streamedcsv search commands has changed. When you execute these search commands after an upgrade, Splunk Enterprise stores and reads the files they create in
$SPLUNK_HOME/var/run/splunk/csv, rather than
The upgrade process moves any existing working files to the new directory and logs the following message to
Creating $SPLUNK_HOME/var/run/splunk/csv and moving inputcsv/outputcsv files into the created directory.
Note the following migration issues:
- Apps, add-ons, or scripts that use the commands or that reference the old working directory could be negatively affected when you upgrade due to the changed directory location.
- You must manually migrate any files that you use in conjunction with
inputcsvthat do not end with the
.csvfile extension, or that are in a subdirectory.
- If you have a component that is external to Splunk Enterprise that uses the
outputcsvcommand, you must manually update the paths of any files or scripts in that component that use the command.
- Additionally, if the component contains files that
outputcsvhas generated, and those files either do not end in
.csvor are in a subdirectory, you must migrate those files to the new working directory manually.
Support for the Deployment Monitor app has been removed
Support for the Splunk Deployment Monitor App has been removed. When you upgrade to Splunk Enterprise 6.3, use the Distributed Management Console (DMC) instead to monitor your distributed deployment. See the Distributed Management Console manual.
Data block signing has been removed
Data block signing has been removed from Splunk Enterprise version 6.2. The feature has been deprecated for some time.
Accelerated custom data model summaries will rebuild on upgrade
When you upgrade to Splunk Enterprise 6.3, any accelerated custom data model summaries that are present on the instance - such as those created by the Splunk App for Enterprise Security - will be rebuilt. This is because of optimizations to data model searches that have been made, which make the searches incompatible with previously generated summaries.
During the rebuild process, CPU, memory, and disk I/O usage on indexers with the summaries will increase significantly. Searches that rely on those data model summaries will be very slow and might not work fully.
If you need to prevent Splunk Enterprise from automatically rebuilding these summaries on upgrade, make the following changes to your Splunk Enterprise configuration before starting an upgrade:
acceleration.manual_rebuilds = true
[tstats] allow_old_summaries = true
There is now a limit on the number of learned source types
For all versions of Splunk Enterprise, the number of source types that an instance can learn in the process of monitoring and indexing files has been limited.
To reduce instances where CPU and memory usage spiked during such operations, a new attribute that controls how many source types an instance learns when it monitors files and analyzes file contents has been created. The limit is 1000, and you can change this setting by editing the following attribute in
limits.conf and restarting Splunk Enterprise:
learned_sourcetypes_limit = <number>
While this setting should prevent memory and CPU spikes, continue to use
inputs.conf to define and apply source types.
Parallel summarization for data model summaries has been enabled
The number of searches that the Splunk platform runs at a time to generate summary files for data models has changed.
When you upgrade to Splunk Enterprise 6.3, the software runs two concurrent search jobs to generate the summary files, instead of one. This change is called "parallel summarization." It might result in an increase in CPU and memory usage on the instance that contains the data models while the search jobs run, but results in faster availability of data model summaries.
You can change this setting back to the previous default for individual data models. See "Parallel summarization" in the Knowledge Manager Manual.
Results for unaccelerated data models now match results from accelerated data models
The way that unaccelerated data models query indexes for events has changed.
These models now query all indexes, rather than just the default index. This means that the number of results you see for unaccelerated data models should now match the number of results you see for accelerated data models.
After you upgrade, you might see more results for an unaccelerated data model than you did prior to upgrading.
You must now enable access to Splunk Enterprise debugging endpoints
Splunk Enterprise used to allow access to debugging endpoints by default. This is no longer the case. When you upgrade, you won't be able to access the debugging endpoints until you make a change in
web.conf and restart Splunk Enterprise:
[settings] enableWebDebug = true
Migration from search head pooling to search head clustering
If you want to migrate to search head clustering from a standalone search head, or from search head pooling, which has been deprecated, you must follow specific instructions and use new Splunk Enterprise instances for search head cluster members. See the following topics in the Distributed Search manual for more information on migrating to search head clustering:
Search head clusters now respect user- and role-based search quotas
When you upgrade to Splunk Enterprise 6.3, any search head clusters that you have deployed will respect and enforce search quotas that are in place for users and roles. This might result in some searches not executing, depending on the number of concurrent searches that are active. To defeat this feature, set the following attributes in
shc_role_quota_enforcement = false shc_local_quota_check = true
The new App Key Value Store service might increase disk space usage
The App Key Value Store (KV Store) service, which provides a way for you to maintain the state of your application by storing and retrieving data within it, might cause an increase in disk usage on the instance, depending on how many apps you run. You can change where the KV Store service puts its data by editing
server.conf, and you can restore data used by KV Store with the
splunk clean CLI command. See "About the app key value store" in the Admin manual.
New installed services open additional network ports
Splunk Enterprise installs and runs two new services: App Key Value Store and App Server. This opens two network ports by default on the local machine: 8065 (for Appserver) and 8191 (for App Key Value Store.) Make sure any firewall you run on the machine does not block these ports. The App Key Value Store service also starts an additional process,
mongod. If needed, you can disable App Key Value Store by editing
server.conf and changing the
dbPath attribute to a valid path on a file system that the Splunk Enterprise instance can reach. See "About the app key value store" in the Admin manual.
Confirm that the introspection directory has the correct permissions
If you run Splunk Enterprise on Linux as a non-root user, and use an RPM to upgrade, the RPM writes the
$SPLUNK_HOME/var/log/introspection directory as root. This can cause errors when you attempt to start the instance later. To prevent this,
$SPLUNK_HOME/var/log/introspection directory to the user that Splunk Enterprise runs as after upgrading and before restarting Splunk Enterprise.
The Splunk DB Connect app can cause issues with data inputs
Due to a design flaw with version 1.1.4 of the Splunk DB Connect app, the "Forwarded Inputs" section of the "Data Inputs" page disappears if you upgrade a Splunk Enterprise instance with the app installed. To work around the problem, upgrade the app to version 1.1.5 before starting an upgrade.
The Splunk Web visualizations editor changes take precedence over existing 'rangemap' configurations for single-value visualizations
If you use the
rangemap search command to define ranges and colors for single-value visualizations on dashboards, use the Format editor instead when you upgrade. Changes that you make with the Format editor to these visualizations override the
rangemap configurations. Going forward, generate new single value visualizations by using a query that does not contain the
rangemap command, and then use the Format editor to configure ranges, colors, or any additional settings.
Any changes that you make with the editor to single-value visualizations that were generated with
= rangemap override edits that you make to the
range map command. Additionally, while the editor attempts to preserve the existing configuration, it no longer recognizes
rangemap as a valid command to generate these types of visualizations.
Formatting for single-value visualizations has changed
The formatting for single-value visualizations has changed in that these visualizations have been redesigned to be as readable as possible from a distance. When you upgrade, dashboards that use these visualizations might be impacted by very large letters or numbers.
To work around the problem, you can either:
- Make use of the new time context if you show a numeric value that you can query over time.
- Use Simple XML to reduce the single value panel height from its default of 115 pixels. Or,
- Replace the single value panel with a custom HTML panel.
See this post on Splunk Answers for additional information prior to upgrading.
New default values for some attributes can impact Splunk operations over SSL
There are new defaults which can possibly impact running Splunk Enterprise over SSL:
supportSSLv3Onlyattribute, which controls how Splunk Enterprise handles SSL clients, now has a default setting of
true. This means that only clients who can speak the SSL v3 protocol can connect to the Splunk Enterprise instance.
cipherSuiteattribute, which controls the encryption protocols that can be used during an SSL connection, now has a default setting of
TLSV1+HIGH:@STRENGTH. This means that only clients that possess a Transport Layer Security (TLS) v1 cipher with a 'high' encryption suite can connect to a Splunk Enterprise instance.
Login page customization is no longer available
Login page customization is no longer available as of version 6.2 of Splunk Enterprise. You can only modify the header and footer of the login page after an upgrade.
The Windows host monitoring input no longer monitors application state
The Windows host monitor input has been modified to no longer monitor the state of installed applications.
Due to a bug in the system call that Splunk Enterprise uses to monitor application state, the Windows Installer service attempts to reconfigure all installed applications.
When you upgrade, any Windows host monitoring input stanzas that reference the "Application" attribute will no longer function. To get application state data, use the Windows Event Log monitor and search for Event ID Nos. 11707 (for installation) or 11724 (for uninstallation/removal.)
It may also be possible to use a powershell script (Get-WmiObject -Class Win32_Product | Format-List -Property Name,InstallDate,InstallLocation,PackageCache,Vendor,Version,IdentifyingNum) or WMIC (wmic product get name,version,installdate).
New installation and upgrade procedures
Beginning with Splunk Enterprise v6.3, the Windows version of Splunk Enterprise has a more streamlined installation and upgrade workflow. The installer now assumes specific defaults (for new installations) and retains existing settings (for upgrades) by default. To make any changes from the default on installations, you must check the "Customize options" button. During upgrades, your only option is to accept the license agreement. See "Installation options."
This feature was introduced in Splunk Enterprise 6.2, but we retain it here for those who upgrade to 6.3 from earlier versions.
The Splunk Web service installs but does not run
Beginning with Splunk Enterprise v6.2, the
splunkd service handles all Splunk Web operations. However, on Windows instances, the installer still installs the
splunkweb service, although the service quits immediately on launch when operating in normal mode. You can configure the service to run in legacy mode by changing a configuration parameter in
web.conf. See "Start Splunk Enterprise on Windows in legacy mode" in the Admin manual.
Important: Do not run Splunk Web in legacy mode permanently. Use legacy mode to temporarily work around issues introduced by the new integration of the user interface with the main splunkd service. Once you correct the issues, return Splunk Web to normal mode as soon as possible.
This change was introduced in Splunk Enterprise 6.2, but we retain it here for those who upgrade to 6.3 from earlier versions.
No support for enabling Federal Information Processing Standards (FIPS) after an upgrade
There is no supported upgrade path from a Splunk Enterprise system with enabled Secure Sockets Layer (SSL) certificates to a system with FIPS enabled. If you need to enable FIPS, you must do so on a new installation.
The default behavior for translating security identifiers (SID) and globally unique identifiers (GUIDs) when monitoring Windows Event Log data has changed
etc_resolve_ad_obj attribute, which controls whether or not Splunk Enterprise attempts to resolve SIDs and GUIDs when it monitors event log channels, is now disabled by default for all channels. When you upgrade, any
inputs.conf monitor stanzas that do not explicitly define this attribute will no longer perform this translation.
Learn about known upgrade issues
To learn about any additional upgrade issues for Splunk Enterprise, see the "Known Issues - Upgrade Issues" page in the Release Notes.
How to upgrade Splunk Enterprise
Upgrade your distributed Splunk Enterprise environment
This documentation applies to the following versions of Splunk® Enterprise: 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14