Manage a rolling restart in Splunk Cloud Platform

Some configuration updates can cause the indexers in your Splunk Cloud Platform deployment to begin a process called a rolling restart. To minimize the impact of a rolling restart, deploy these updates during off-peak hours.

What users experience during a rolling restart

A rolling restart is a sequential restart of Splunk indexers that allows indexing to continue during the restart process.

While indexing remains available at all times during a rolling restart, non-Splunk clients that do not follow best practices for retrying connections and managing backpressure might be impacted by an individual node restarting. Using forwarders or other types of load balancers, rather than network inputs alone, increases the robustness of your indexing during a rolling restart.

Searches still run during a rolling restart, but they might return incomplete results. Users running searches in Splunk Web receive a message warning of incomplete search results.

What triggers a rolling restart

Deploying certain changes triggers a rolling restart. Examples of changes that trigger a rolling restart include, but are not limited to, the following tasks:

• Deleting the last HEC token (which deletes the app, causing a rolling restart).
• Installing some apps and add-ons. See Restart versus reload behavior of common apps and .conf files.

Deploying a seemingly safe change can indirectly trigger a rolling restart. For example, adding an index doesn't trigger a restart by itself. But if you or another admin has made other changes that trigger a rolling restart and not deployed them, then when you deploy your change that adds an index, you also deploy the previous changes that trigger the rolling restart.

Restart versus reload behavior of common apps and .conf files

Most configuration files do not trigger a restart when configuration changes occur, but instead trigger a less time-consuming file reload. To minimize service disruptions, before you install apps or deploy configuration updates in Splunk Cloud Platform, consider the restart versus reload behavior of relevant apps and configuration files.

For more information on configuration file reload behavior, see Configuration file reload triggers in app.conf.

The following tables list some common apps and configuration files and show whether they trigger a restart or a reload.

• Reload or restart behavior of common .conf files
• Reload behavior of common apps

Reload or restart behavior of common .conf files

Most Splunk configuration files are now reloadable. The following table shows the reload or restart behavior of some frequently used configuration files in Splunk Cloud Platform:

.conf file name	Used for	Reload or restart
authorize.conf	This file is used to configure roles and granular access controls.	reload
collections.conf	This file is used to configure KV store settings for a given app.	reload
distsearch.conf	This file is used to configure attributes and values you can use to configure distributed search.	reload
indexes.conf	This file is used to configure indexes and their properties. For a list of specific changes to this file that require a restart, see Determine which indexes.conf changes require a restart in the Splunk Enterprise documentation.	reload/restart
inputs.conf	This file is used for HEC CRUD operations, configuring tcp ports for forwarders, configuring scripted inputs for apps, and configuring file system monitoring. Splunk Cloud Platform supports stanza-level reload for inputs.conf. For more information on stanza-level reload, including a list of reloadable stanzas, see Stanza-level reload triggers for inputs.conf.	reload/restart
multikv.conf	This file is used to configure multikv rules for extracting events from table-like events, such as the output of top, ps, ls, netstat, etc.	reload
props.conf	This file is used to set indexing property configuration, including timezone offset, custom source type rules, and pattern collision properties. Also, map transforms to even properties.	reload
restmap.conf	This file is used to create custom REST endpoints.	reload
server.conf	This file is used to configure which settings should be replicated within a search head cluster. Changes to the [shclustering] stanza require reload only. All other changes to server.conf require a restart.	reload/restart
transforms.conf	This file is used to configure regex transformations to perform on data inputs. Use in tandem with props.conf.	reload
ui-tour	This file is used to configure in-product tours of Splunk software features.	reload
web.conf	This file is used to configure tcp port to listen to incoming connections, appserverports, connectiontimeout.	reload
wmi.conf	This file is used to configure access to Windows Management Instrumentation (WMI).	reload

Reload behavior of common apps

The following table shows the reload behavior of frequently used apps and add-ons in Splunk Cloud Platform:

This list pertains to the specified version of each app. Changes made to an app's configuration settings in subsequent app versions might trigger a restart instead of a reload.

App name	Version	Used for	Reload
Cisco Networks Add-on for Splunk Enterprise	2.5.8	This add-on sets the correct sourcetype and fields for identifying data from Cisco IOS, IOS XE, IOS XR, NX-OS devices in Splunk® Enterprise.	reload
Force Directed App For Splunk	3.0.1	The Force Directed App For Splunk helps you graph out attack paths and review links in your data. Built on D3 this app will allow you to search any form of data that has a source and target.	reload
Lookup File Editor	3.3.2	This app provides an Excel-like interface for editing, importing, and exporting lookup files (both KV store and CSV based lookups)	reload
Palo Alto Networks Add-on for Splunk	6.1.1	This add-on collects and correlates data from Firewalls, Panorama, Traps Endpoints, Aperture SaaS Security, AutoFocus, MineMeld, and WildFire.	reload
Palo Alto Networks App for Splunk	6.1.1	This app combines Palo Alto Networks security platform features with Splunk's investigation and visualization capabilities to provide advanced security reporting and analysis.	reload
Python for Scientific Computing (for Linux 64-bit)	1.4	This add-on contains a Python interpreter bundled with the following scientific and machine learning libraries: numpy, scipy, pandas, scikit-learn, and statsmodels. With this add-on, you can import these powerful libraries in your own custom search commands, custom rest endpoints, modular inputs, and so forth.	reload
Punchcard Custom Visualization	1.3.0	This Punchcard Custom Visualization app provides interactive ways to visualize and investigate cyclical trends in your data.	reload
Qualys Technology Add-on (TA) for Splunk	1.4.3	This add-on provides pre-built inputs for Qualys Cloud Platform data.	reload
Splunk Add-on for Amazon Web Services	4.6.0	This add-on lets Splunk admins collect data from AWS accounts, including configuration details, EC2 instance and EBS metadata, compliance information, CloudWatch log data, performance and billing metrics, S3 bucket stats, and more.	reload
Splunk Add-on for Cisco ASA	3.4.0	The Splunk Add-on for Cisco ASA allows a Splunk software administrator to map Cisco ASA devices, Cisco PIX, and Cisco FWSM events to the Splunk CIM.	reload
Splunk Add-on for Microsoft Cloud Services	3.1.0	This add-on lets Splunk admins pull activity logs, service status, operational messages, Azure audit, Azure resource data and Azure Storage Table and Blob data from a variety of Microsoft cloud services using the Office 365 Management APIs, Azure Service Management APIs and Azure Storage API.	reload
Splunk Add-on for Microsoft Office 365	1.1.0	This add-on lets Splunk admins pull service status, service messages, and management activity logs from the Office 365 Management API.	reload
Splunk Add-on for Microsoft Windows	6.0.0	This add-on provides predefined inputs to collect data from Windows systems and maps data to the Common Information Model.	reload
Splunk Add-on for Unix and Linux	6.0.2	The Splunk Add-on for Unix and Linux allows a Splunk software administrator to collect *nix data from *nix hosts.	reload
Splunk App for AWS	5.1.3	This app provides insight into your Amazon Web Services account. The app includes pre-built dashboards, reports, and alerts that provide real-time visibility into your AWS environment, including your AWS Config, CloudWatch, CloudTrail, Billing, S3, VPC Flow Log, Amazon Inspector, and Metadata inputs.	reload
Splunk App for Windows Infrastructure	1.5.2	This app provides pre-built data inputs, searches, reports, and dashboards that let you monitor, manage, and troubleshoot Windows operating systems, including Active Directory elements, from a single location.	reload
Splunk Common Information Model (CIM)	4.13.0	This add-on contains a collection of pre-configured data models that support the consistent, normalized treatment of data for maximum efficiency at search time.	reload
Splunk Dashboard Examples	7.3.0	The Splunk Dashboard app delivers examples that give you a hands-on way to learn the basic concepts and tools needed to rapidly create rich dashboards using Simple XML.	reload
Splunk Datasets Add-on	1.0	This app delivers new SPL commands, custom visualizations, assistants, and examples to explore a variety of machine learning concepts.	reload
Splunk Machine Learning Toolkit	5.2.0	This add-on provides an intuitive interface to build, edit and analyze table datasets (tables) without SPL.	reload
Splunk Sankey Diagram - Custom Visualizations	1.5.0	Sankey diagrams show metric flows and category relationships. You can use a Sankey diagram to visualize relationship density and trends.	reload
Splunk Supporting Add-on for Active Directory	2.2.1	This app provides support functions to the Windows Infrastructure, Active Directory, and Exchange apps that enable you to extract information from an Active Directory database.	reload
Splunk Timeline - Custom Visualization	1.4.0	A timeline visualization shows activity time intervals and discrete events for a resource set.	reload

Guidance for managing a rolling restart

To minimize impact to users, deploy configuration changes during times that are off peak for both indexing and searching. You can identify off-peak times from the Snapshots in your Splunk Cloud Monitoring Console. See Monitor your Splunk Cloud Platform Deployment.

During a rolling restart, monitor indexing and search performance with the Splunk Cloud Monitoring Console.

More information

For more information about how a rolling restart works, see Perform a rolling restart of an indexer cluster in the Splunk Enterprise documentation. Note that some of the advanced options are not available by default in Splunk Cloud Platform.