Known issues

The following are issues and workarounds for this version of Splunk Enterprise.

Issues are listed in all relevant sections. Some issues appear more than once.

Refer to System requirements in the Installation Manual for a list of supported platforms and architectures.

For a list of deprecated features and platforms, refer to Deprecated features and removed features in this manual.

Upgrade issues

Date filed	Issue number	Description
2020-08-31	SPL-194426	External search command chunked v2 python SDK fails with multibyte result data under python 3. Workaround: Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters. App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps. Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available.
2020-07-10	SPL-191850	The .deb installation package will fail if dpkg version doesn't support an .xz compressed control file. Workaround: Update dpkg to version 1.17.6 or later.
2018-04-13	SPL-153403	After running the "clean userdata" command, admin is unable to login with msg "No users exist. Please set up a new user." Workaround: Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk [user_info] PASSWORD = <yourpassword>

Data input issues

Date filed	Issue number	Description
2024-01-13	SPL-249543, SPL-251748, SPL-251749, SPL-253929, SPL-251746, SPL-253927, SPL-253928	TcpInputProcessor not able to drain splunktcpin queue during graceful shutdown. Workaround: Splunk recommends customers set `useACK` to true to ensure in-memory is not dropped in the event of indexer rolling restarts or repaving. Thus, the best short-term solution is to set `useACK` to `true`.
2024-01-10	SPL-249424, SPL-249409	Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x
2024-01-10	SPL-249423, SPL-249409	Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x
2024-01-10	SPL-249422, SPL-249409	Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x
2023-11-07	SPL-246769, SPL-243845	HTTP Input HEC input ignores _meta in inputs.conf
2023-11-07	SPL-246770, SPL-243845	HTTP Input HEC input ignores _meta in inputs.conf
2023-11-07	SPL-246768, SPL-243845	HTTP Input HEC input ignores _meta in inputs.conf
2023-06-15	SPL-241076, SPL-251249, SPL-251251, SPL-251329, SPL-251250	Metrics event can be indexed in default event index when mcollect is used. Workaround: Avoid to restart if queue is blocked. (wish maybe should be best practise but the revert is what always happen customer queue blocked let restart to solve it .)
2022-08-17	SPL-228646, SPL-228645	Restart is needed when AWS access key pairs rotate (w/o grace period) or other S3 config settings for Ingest Actions become invalid
2022-04-08	SPL-222366	Ingest Actions does not work with Splunk Free, Personalized Devtest, Developer, and Forwarder-only licenses

Search issues

Date filed	Issue number	Description
2023-12-18	SPL-248552	ProcessDispatchedSearch error displayed - The process cannot access the file because it is being used by another process Workaround: There is no workaround. Details This issue affects Microsoft Windows environments only. The only impact of this error is that some log lines might be dropped from Splunk's own search.log and splunkd.log files. Searches still work as usual.
2023-12-12	SPL-248297	Higher memory usage than usual on Windows systems after upgrade from Splunk Enterprise version 9.0/8.x to version 9.1.x/9.2.x Workaround: As of Splunk Enterprise version 9.1.x, a new system call for allocating memory called mimalloc has been implemented on Windows. Prior to Splunk Enterprise version 9.1, the Splunk Enterprise implementation for Windows used a default system call. The new mimalloc system call can cause an increase in memory usage, which impacts search heads, indexers, and universal forwarders, especially when data model acceleration is in use. This issue will be fixed in Splunk Enterprise versions 9.1.5 and 9.2.2. Upgrade to Splunk Enterprise version 9.1.5 or 9.2.2 when they become available.
2023-10-27	SPL-246383, SPL-246534, SPL-246535, SPL-246536, SPL-246537, SPL-246538, SPL-246539	Excessive logging in AuditLogger of "action=admin_all_objects, info=denied" after upgrade from 9.0.4 to 9.1.1
2023-10-06	SPL-245471, SPL-239942	Splunk fails to finalize a search on smart store enabled cluster when phased_execution_mode is set to multithreaded
2023-10-04	SPL-245341, SPL-239942	Fix the issue that splunk search cannot be finalized when indexer is slow
2023-10-03	SPL-245287, SPL-245166	UTF-8 characters should not be hex-encoded in Splunk logs. Workaround: 1. Identify the specific log files that are improperly escaping UTF-8 characters. 2. For each log file, find the appender name corresponding to the log file in $SPLUNK_HOME/etc/log.cfg. For example, the splunkd.log log file is defined by this appender: appender.A1.fileName=${SPLUNK_HOME}/var/log/splunk/splunkd.log 3. Add a new line in $SPLUNK_HOME/etc/log.cfg as follows corresponding to the appender name: appender.<appender name>.escape=false For example, for the A1 appender which governs splunkd.log, add the line: appender.A1.escape=false 4. Restart Splunk.
2023-09-27	SPL-245135, SPL-245127	Indexer Search crash with no back-trace in PCRE2 on X86_64 Workaround: Re-running the search is the only workaround. This issue is not deterministic within a search, so re-running will usually work.
2023-09-25	SPL-245065, SPL-225303, SPL-245054	Segfault in BucketSummaryActorThread, originating in the lookup processor, causes crashing search processes for Data Model Accelerations Workaround: Update the `max_memtable_bytes` in my the limits.conf/[lookup] stanza to a large value, such as 2147483648 (2GB), to increase the threshold at which lookup tables are indexed to disk on the indexers instead of using in-memory indexes for those lookups. We recommend sizing the threshold based on lookups in your environment. Please note that this will increase the amount of memory used for searches that use lookups, which could be significant in environments with many concurrent searches.
2023-09-19	SPL-244841, SPL-244119	Search processes segfault'ing due to Logger appender without layout definition
2023-06-09	SPL-240774	The DELIMS setting or the kvdelim option may not be applied correctly when the k/v delim character appears 2 or more times in a field value Workaround: Perform field extractions by modifying your searches using other commands, such as the rex command or eval command.
2023-04-14	SPL-238738	Federated Search for Splunk does not support the "Show Source" Field Action
2023-03-28	SPL-237902	Ad hoc searches that specify earliest relative time offset assuming from 'now' should explicitly include 'latest=now' to avoid a potential time range inaccuracy Workaround: Ad hoc searches searches that use the earliest time modifier with a relative time offset should also include 'latest=now' in order to avoid time range inaccuracies. For example, if you want to get all events from the last 10 seconds starting at 01:00:10, the following search returns all events that occur between the time of 01:00:00 and 01:00:10, as expected. index=main earliest=-10s latest=now Running the same search without including 'latest=now' might produce unpredictable results or impact performance in certain scenarios when the search head is overloaded with ad hoc searches. See Specify earliest relative time offset and latest time in ad hoc searches in the Splunk platform Search Manual.
2022-10-20	SPL-231830, SPL-239319, SPL-239320	SearchJob sometimes fails and returns error "Search <ID> not found. The search may have been cancelled while there are still subscribers" Workaround: Remark : this Splunk Enterprise issue may impact ITSI UI with loading issues (KPI and thresholds preview, Share Base search validation, Entity import, Maintenance windows preview) Fix : upgrade Splunk to a fixed version
2022-07-29	SPL-227633	Error : Script execution failed for external search command 'runshellscript' Workaround: The setting precalculate_required_fields_for_alerts=0 can be set on saved searches that have no other alert actions attached aside from the "Run A Script" action, to quash the error. For saved searches that have multiple alert action attached, this may not be safe as it will disable back propagation of required fields for all alert actions, which might result in the parent search extracting more fields than required, which could negatively impact performance for that search.
2021-12-21	SPL-216787	Searches are cancelled or time out when the user leaves the browser window or switches tabs. Workaround: Change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62. Details This issue is caused by power saving settings in recent browser versions, where Javascript timers may be throttled. The user typically sees the following message in the search window on foreground searches: DAG Execution Exception: Search has been cancelled Search auto-canceled The search job has failed due to an error. You may be able to view the job in the Job Inspector
2021-09-22	SPL-212495, SPL-196040, SPL-219811	Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles Workaround: none
2021-02-25	SPL-201628	`srchTimeWin` and `srchTimeEarliest` settings cannot be unset for the admin role. Workaround: Ensure that the admin role is not configured as "Unset" and is explicitly configured to either no restriction or a restriction in the UI (Navigate to Edit Role > Resources > Role search time window limit), or through conf file authorize.conf under attribute name srchTimeEarliest.
2020-12-04	SPL-198284, SPL-231587	Crash in search process in PrecacheUsersThread when max_searches_per_process is set lower than default Workaround: Set limits.conf back to default, by removing any override of max_searches_per_process. For example: [search] max_searches_per_process=1 to [search]
2020-08-31	SPL-194426	External search command chunked v2 python SDK fails with multibyte result data under python 3. Workaround: Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters. App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps. Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available.
2020-02-12	SPL-183259	When generating LISPY for field values that are numbers (""), the values aren't deduplicated, which can cause slowdowns in certain scenarios Workaround: Dedup values in search before, for example: instead of index="field_test" [search index="field_test" globalCallID_callId=1234* | fields globalCallID_callId] add a stats or dedup in the subsearch: index="field_test" [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ] If that list is still large and you're seeing the slowdown, consider moving the filtering to a | where after the initial search, for example: index="field_test" globalCallID_callId=* | where [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ]
2020-01-10	SPL-181573	geostats provides incorrect results for lower zoom levels when split BY has a higher cardinality than globallimit. Workaround: - Increase globallimit to the value of "unique values" number mentioned in the warning message: "The split by field <field> has a large number of unique values <number>. Chart column set will be trimmed to 10. Use globallimit argument to control column count." - Use very high globallimit in geostats and post process after if needed - Don't use BY in geostats - Use lower cardinality BY and/or higher globallimit in geostats
2017-07-13	SPL-143111	"Splunkd daemon is not responding" when edit local windows event log collection
2017-04-04	SPL-140765	Splunk having problems extracting json file consisting of 68k plus key-value pairs
2016-11-29	SPL-133182	When two datasets have identical names but one is local (private) while the other is global, attempts to view or extend the global dataset use results from the local dataset instead.
2014-10-02	SPL-91638, SPL-107375	For scheduled searches in a search head cluster, empty search jobs may appear in the job inspector for a cluster member.

Federated search issues

Date filed	Issue number	Description
2024-04-23	SPL-254718, SPL-253248, SPL-255069	Federated searches not completing with error "Socket error during transaction. Socket error: Success"
2024-04-19	SPL-254539, SPL-253986	Transparent Federated Search should not ignore federated service account index permission when fsh user is set to SPLUNK_SYSTEM_USER
2024-04-19	SPL-254540, SPL-253986	Transparent Federated Search should not ignore federated service account index permission when fsh user is set to SPLUNK_SYSTEM_USER
2024-04-05	SPL-253755, SPL-252488, SPL-253757	federated search should alert ( and block the search ) when it is run in realtime mode
2024-03-27	SPL-253248, SPL-254718, SPL-254719, SPL-254720, SPL-254722, SPL-254721	Federated searches not completing with error "Socket error during transaction. Socket error: Success"
2024-03-26	SPL-253175, SPL-244551	Federated search failures seen on RSH due to terminated search connections from FSH
2024-03-13	SPL-252583	standard mode `from` federated data model does not alert if an eventtype is added at the end of the query Workaround: Make sure that standard federatd provider is reachable
2024-03-12	SPL-252488, SPL-248786, SPL-253755	Lookups in transparent mode don't use proper lookup when fsh and rsh have lookup with same name
2024-03-12	SPL-252474	When onPrem FSH's license expires, it still sends federated searches to RSH and then immediately cancels the searches. Workaround: Make sure that standard federatd provider is reachable
2024-03-11	SPL-252400	federated search for Splunk: a wildcard search on a search head that has unreachable federated indexes should not fail Workaround: Make sure that standard federatd provider is reachable
2024-02-23	SPL-251536	Block edit request to change existing federated provider mode and useFSHKO settings
2024-01-18	SPL-249666, SPL-244551	FS-StandardMode : Standalone sub-search with HEAD doesn't return any results
2023-12-21	SPL-248786, SPL-252486, SPL-252487, SPL-252488	Lookups in transparent mode don't use proper lookup when fsh and rsh have lookup with same name Workaround: If the `rsh` is getting transparent searches and it does not have indexers connected to it, the `rsh` does not look on the kvstore values that were sent to it from the `fsh`. The workaround is to have indexers attached to the `rsh`
2023-09-05	SPL-244248, SPL-239298	Federated Search, Enterprise --> Cloud configuration: Performance degradation increases when the number of indexers increases in the RSH Workaround: One possible workaround is to use a more efficient query. For example, use "| tstats count where index=main by splunk_server" instead of "index=main | stats count by splunk_server". Another workaround is to change the max_workers_searchparser setting to a value lower than its default. Use this workaround if you are using your Splunk Enterprise federated search head (FSH) instance only for running federated searches. This workaround might affect non-federated searches. On the Splunk Enterprise FSH, follow these steps: Create limits.conf in a local/ folder. Set the max_workers_searchparser setting to a number lower than its default (1 or 2). For more information about this setting see the Admin Manual. Test which setting value provides a better performance.
2023-08-28	SPL-243968, SPL-239689, SPL-244367	In transparent mode Federated Search for Splunk, custom search commands and the "outputlookup" command should run only on the local deployment. Instead they run on the remote deployment, leading to errors, incorrect results.
2023-08-14	SPL-243209, SPL-241502	Transparent federated search with service account making api calls takes excessive time
2023-07-20	SPL-242282, SPL-242864	Federated Searches fail for union commands when query optimization diverge between FSH x RSH
2023-07-12	SPL-242049, SPL-248189, SPL-248311, SPL-248312	Kvstore files are not converted to csv files in the bundles when local indexers are not present even when remote providers are present Workaround: If an indexer ( a distributed peer ) is added to the local deployment ( the federated search head ), then the issue is resolved. On deployment that don't have indexers at the moment a low performance indexer should be created ( on a vm etc ) and added as a distributed peer.
2023-05-02	SPL-239436	In standard mode federated search, outputlookup existence check on RSH causes search to terminate early although it is not run on RSH Workaround: Define the lookup on both federated search head and remote search head.
2023-04-17	SPL-238767, SPL-244936, SPL-244937	Standard mode federated search with longer than a minute From command searches, encounters socket ReadWrite error when the federated provider points to a cloud Load balancer, due to idle timeout on the LoadBalancer config Workaround: If you encounter this issue, update the federated provider definition (created on the federated search head in Splunk Web), so that its Remote Host points to a remote deployment cluster member instead of to the remote deployment cluster load balancer.
2023-04-14	SPL-238738	Federated Search for Splunk does not support the "Show Source" Field Action
2023-04-11	SPL-238512, SPL-239266	Federated search UI does not support mapping federated indexes to data model datasets that have dot characters in their names
2023-04-10	SPL-238501	Federated search "outputlookup" command cannot add data to local lookup table Workaround: Define the same lookup on the remote search head too, so the remote search head will not error out early and return 0 results.
2023-03-24	SPL-237796, SPL-248319	In transparent mode Federated Search for Splunk, the makeresults command returns more rows than expected Workaround: Convert all occurrences of makeresults to makeresults | head 1. If you need more results, change the head command parameter accordingly. For example,makeresults count=5 would become makeresults count=5 | head 5.
2022-10-19	SPL-231712	Create/Edit Role - In the UI, the "Wildcards" tool cannot be used to specify allowed federated indexes for standard mode federated search
2022-07-15	SPL-226877	Federated Search UI Error: Cannot create saved search dataset for federated index if dataset name contains space Workaround: Use REST API to create the federated saved search instead: curl -k -u <username>:<password> -X POST https://localhost:8089/servicesNS/nobody/search/data/federated/index -d name=federated:index_kathy -d federated.dataset='savedsearch:ss with space' -d federated.provider=remote_deployment_1 See Federated search endpoint descriptions in the REST API Reference Manual.
2022-05-31	SPL-225037	Remote dataset dropdown menu resets to "Index" after selecting federated provider
2022-02-25	SPL-219793	Some commands in federated searches return incorrect resultCount values when run in verbose mode Workaround: Use Verbose and Smart mode specifically for searches with transforming commands like stats, chart, and timechart, and then review the results in the Statistics tab. To review event counts, run non-transforming searches in Fast mode. Search-time field extraction usually requires searches without transforming commands that run in either Verbose or Smart mode. When you run searches in Fast mode, you can ensure that search-time field extraction takes place for federated searches by appending | fields * to the ends of your searches.
2022-02-08	SPL-218842, SPL-252272, SPL-242740	Some reporting commands in federated search return incorrect eventCount Workaround: Use Verbose and Smart mode specifically for searches with transforming commands like stats, chart, and timechart, and then review the results in the Statistics tab. To review event counts, run non-transforming searches in Fast mode. Search-time field extraction usually requires searches without transforming commands that run in either Verbose or Smart mode. When you run searches in Fast mode, you can ensure that search-time field extraction takes place for federated searches by appending | fields * to the ends of your searches.
2022-02-08	SPL-218841	Reporting command in verbose mode returns 0 events despite correct event_count
2021-10-14	SPL-213745, SPL-251131	Standard mode federated search: Unable to set federated index as default index

Saved search, alerting, scheduling, and job management issues

Date filed	Issue number	Description
2024-01-19	SPL-249712	Emails from alerts not sent when using sendemail due to authentication error caused by changes in cli_common library Workaround: Replace Python scripts for sendemail with older versions, specifically sendemail.py and sendemailhandler.py. The Python scripts are located in the Search and Reporting app under $SPLUNK_HOME/etc/apps/search/bin. Run a test search to confirm emails are NOT arriving. Confirm/update email settings in one UI (email config, ensure linkhostname is blank, ensure there is no reply email is in the field "send as"). Allow time for replication. Confirm email settings in UI on each box (email config, ensure linkhostname is blank, ensure there is no reply email is in the field "send as"). Backup 9.1.x sendemail Python scripts on each host. Remove 9.1.x sendemail Python scripts on each host. Restore 9.0.x backup sendemail Python scripts on each host. Run a test search to confirm emails are arriving on each host.
2023-11-08	SPL-246785, SPL-244383	Search-Scheduler Splunk Crashes on Job Servers in SHC. Workaround: Workaround collections.conf [LoggedOutSessionTokens] disabled = true server.conf [general] invalidateSessionTokensOnLogout = false
2023-09-20	SPL-244951, SPL-246533, SPL-246542, SPL-267236	Frequent deferred and skipped searches when making changes to savedsearches configurations in Search Head Cluster
2023-07-21	SPL-242301, SPL-231558	The UI trigger for summary rebuild doesn't work for some accelerated data models that have no root-event dataset and have a reporting command in first root search dataset Workaround: The workaround is to change the Data Model definition to reorder the root search objects such that the root search object that can be accelerated is the very first one in the list. For instance, for the provided `test_internal_audit_logs.json`, edit the JSON file on disk and move `failed_searches` dataset before `fully_completed_searches`.
2023-07-07	SPL-241821	Data Model Accelerations that have Automatic Rebuilds enabled may lead to unbounded memory growth due to search expansion, resulting in Out of Memory errors Workaround: For a data model that is experiencing high memory usage, perform the following steps: On your Splunk platform deployment, in Splunk Web, select Settings and then Data Models. Select Edit for the data model that is experiencing high memory usage, and then select Edit Acceleration. Open Advanced Settings. Disable Automatic Rebuilds. Furthermore, applying index constraints to restrict the list of indexes searched for building a given DMA summary and applying tags allowlisting would help curtail the memory usage.
2019-09-20	SPL-176812	Multiple SH Clustering with single deployer can't use datamodel summary sharing
2018-09-19	SPL-160286	The data preview for the Add Data workflow does not display for Log to Metrics source types
2017-11-29	SPL-146802	Distributed environment requires index defined on search head for log event alerts
2017-08-14	SPL-143947	Report acceleration is broken for users with a configured role-based access filter

Charting, reporting, and visualization issues

Date filed	Issue number	Description
2023-09-18	SPL-244788, SPL-247096, SPL-247097	"Awaiting user confirmation" error when img src is a token that is set to a URL after SXML dashboard loads Workaround: If possible, hard code the img src domain and capture the rest of the img URL as a token. For example, instead of `<img src="$image_url$">`, use `<img src="https://www.example.com/$image_url$">` and remove the domain from the value of $image_url$. Another option is to add a hidden placeholder image to the HTML panel with the issue: <style> .placeholder-img { display:none } </style> <img class="placeholder-img" src="https://dev.splunk.com/test.jpg" />
2023-07-19	SPL-242232	Dashboard Studio - CSV export does not wrap string values with quotes Workaround: Export CSV by opening in Search first and then exporting it from the Search page.
2023-07-05	SPL-241761	Dashboard Studio - Table view export does not include all the data of the table in the CSV Workaround: Export table rows by opening in Search instead of selecting Export. Image: SelectSearchvrsExportTableView.png
2023-06-30	SPL-241687, SCP-62360	In Dashboard Studio table visualizations, the number 0 is on the left side of cells, but all other numbers are on the right side.
2023-06-29	SPL-241626	Chain searches connected to a base search with a risky command don't provide a pop-up to accept the risky command. Workaround: Create another visualization, such as a rectangle, and connect it with only the base search. Accept the risky command activated by the base search.
2023-06-21	SPL-241274	Dashboard Studio fails to load dashboards and displays the error "Cannot convert undefined or null to object" when search results return "null" values. Workaround: Replace the "null" value with "empty" by appending an SPL replace command to the search query. For example, | replace "null" WITH "empty" IN <fieldname>.
2023-06-14	SPL-240966, SPL-241284, SPL-241285	Classic Simple XML dashboards parsing error: "Cannot convert undefined or null to object" Workaround: NA
2023-06-14	SPL-240964	Visualization action buttons are not working as expected in Dashboard Studio home dashboards. Workaround: Open the impacted dashboard in its original dashboard page instead of the home dashboard page.
2023-06-14	SPL-240965	Dashboard Studio home dashboard flickers on specific viewport resolutions with scrollbar visibility set to always Workaround: Resize the viewport or current monitor resolution. If the issue persists, try changing the dimensions of the dashboard.
2023-06-08	SPL-240750	Inconsistency in displayed timezone in Dashboard Studio when using time range tokens
2023-05-18	SPL-240082, SPL-241349, SPL-241350	Classic Simple XML dashboards with HTML panels using external images and tokens show the error "Awaiting User Confirmation". Workaround: Remove either the external images or the tokens from the HTML panel.
2023-04-21	SPL-239070	SVGs in Dashboard Studio have white background instead of transparent in dark mode
2016-04-27	SPL-118911	In SHC, referenced saved real-time searches in a dashboard do not stream results. Workaround: See Troubleshoot referenced real-time searches for workaround details.

Distributed search and search head clustering issues

Date filed	Issue number	Description
2021-09-22	SPL-212495, SPL-196040, SPL-219811	Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles Workaround: none
2021-03-26	SPL-203060	The splunkd process changes the local distsearch.conf on service start Workaround: There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will: Remove any settings that define default values already set in the /default/distsearch.conf file. Removes comments preceded by a hash. Reorders the KV pairs alphanumerically within a stanza. Reorders stanzas within the file.
2017-11-29	SPL-146802	Distributed environment requires index defined on search head for log event alerts
2017-03-13	SPL-138654	Splunk searches fail when filepath gets too long on Windows
2016-07-12	SPL-124085	On Search Head Cluster It is not possible to remove an App from the SHs once it has been disabled.

Data model and pivot issues

Date filed	Issue number	Description
2023-07-21	SPL-242301, SPL-231558	The UI trigger for summary rebuild doesn't work for some accelerated data models that have no root-event dataset and have a reporting command in first root search dataset Workaround: The workaround is to change the Data Model definition to reorder the root search objects such that the root search object that can be accelerated is the very first one in the list. For instance, for the provided `test_internal_audit_logs.json`, edit the JSON file on disk and move `failed_searches` dataset before `fully_completed_searches`.
2023-07-07	SPL-241821	Data Model Accelerations that have Automatic Rebuilds enabled may lead to unbounded memory growth due to search expansion, resulting in Out of Memory errors Workaround: For a data model that is experiencing high memory usage, perform the following steps: On your Splunk platform deployment, in Splunk Web, select Settings and then Data Models. Select Edit for the data model that is experiencing high memory usage, and then select Edit Acceleration. Open Advanced Settings. Disable Automatic Rebuilds. Furthermore, applying index constraints to restrict the list of indexes searched for building a given DMA summary and applying tags allowlisting would help curtail the memory usage.
2019-09-20	SPL-176812	Multiple SH Clustering with single deployer can't use datamodel summary sharing

Indexer and indexer clustering issues

Date filed	Issue number	Description
2024-05-29	SPL-256658, SPL-255517	Indexer Discovery deadlock during tcpout reload
2024-05-29	SPL-256657, SPL-255517	Indexer Discovery deadlock during tcpout reload
2024-04-04	SPL-253649, SPL-246435	Rolling Restart generate fix-up task that search factor taking hours complete as search files replication fails.
2024-01-13	SPL-249543, SPL-251748, SPL-251749, SPL-253929, SPL-251746, SPL-253927, SPL-253928	TcpInputProcessor not able to drain splunktcpin queue during graceful shutdown. Workaround: Splunk recommends customers set `useACK` to true to ensure in-memory is not dropped in the event of indexer rolling restarts or repaving. Thus, the best short-term solution is to set `useACK` to `true`.
2016-08-25	SPL-127353	Data rebalance finishes early when one peer is the source for all buckets Workaround: when only one indexer in a cluster indexed data (has all the searchable copies), rebalance once before adding the new indexer, and then rebalance a second time

Universal forwarder issues

Date filed	Issue number	Description
2024-02-22	SPL-251517, SPL-237849	CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 [1] Upgrade to version 9.0.0.1 issue doesn't appear. [2] The detailed workaround didn't solve the issue.
2024-02-22	SPL-251515, SPL-237849	CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 [3] Upgrade to version 9.0.0.1 issue doesn't appear. [4] The detailed workaround didn't solve the issue.
2024-02-22	SPL-251516, SPL-237849	CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 [5] Upgrade to version 9.0.0.1 issue doesn't appear. [6] The detailed workaround didn't solve the issue.
2023-12-14	SPL-248479, SPL-253092	9.1.1 HF enters state of constant blocking due to broken S2S protocol Workaround: To fix the issue, customers will have to add queueSize on the Splunk HF/IHF instance where persistent queue is enabled, in inputs.conf under the same stanza where persistentQueueSize is set. inputs.conf [<input stanza>] persistentQueueSize=<no change. keep existing value> queueSize=<100MB or 1% of total system memory(whichever is less)>
2023-09-08	SPL-244414	Crashing in TcpOutEloop thread after upgrade from 9.1.x
2023-07-13	SPL-242093, SPL-242240	At upgrade, the Linux RPM/DEB installer creates a default "splunkfwd" account, causing issues when Splunk is already managed by another account Workaround: See https://docs.splunk.com/Documentation/Forwarder/9.1.1/Forwarder/Installleastprivileged to manage the splunkfwd user.
2022-08-17	SPL-228646, SPL-228645	Restart is needed when AWS access key pairs rotate (w/o grace period) or other S3 config settings for Ingest Actions become invalid
2022-06-23	SPL-226019	Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality.
2022-06-06	SPL-225379	Ownership of files mentioned in manifest file is splunk:splunk instead of root:root after enabling boot start as root user for initd Workaround: When changing UF user, manually chown SPLUNK_HOME to the new user, including first time install/upgrade, or manually enable boot-start.
2022-03-23	SPL-221239	System Introspect App fails when universal forwarder is installed at non-admin user

Monitoring Console issues

Date filed	Issue number	Description
2024-06-28	SPL-258394	Health Report for destination output issues show Last 50 detailed logs in Indexer Cluster nodes but not in Search Head or Cluster Manager
2023-12-08	SPL-248160, SPL-243155	The Monitoring Console does not show Indexer and Search Head statistics on the Overview page
2021-03-29	SPL-203100	Summary page on monitoring console doesn't show correct RF/SF when not running on the CM.
2019-11-13	SPL-179528	The splunktcp and splunktcp-ssl stanzas are not reloadable in inputs.conf
2017-08-14	SPL-143981	Uninstall app dialog does not show the app name correctly when the app doesn't have the label
2017-05-24	SPL-141982	Upload modal should use size=large File element
2017-04-19	SPL-141274	Clicking Install multiple times in Install dialog causes error
2016-11-14	SPL-132151	XML error when trying to download uninstalled app

Splunk Web and interface issues

Date filed	Issue number	Description
2023-08-08	SPL-242943, SPL-242699, SPL-246958, SPL-246959	Unable to load app list after upgrading to 9.1 using the Free license
2023-06-30	SPL-241690, SPL-240758	"File Integrity checks found 4281 files that did not match the system-provided manifest." shows in message but does not appear in the "Integrity Check of Installed Files" dashboard. Workaround: Clear the notification to dismiss the error and it will not appear again.
2023-06-30	SPL-241691, SPL-240758	"File Integrity checks found 4281 files that did not match the system-provided manifest." shows in message but does not appear in the "Integrity Check of Installed Files" dashboard. Workaround: Clear the notification to dismiss the error and it will not appear again.
2023-06-30	SPL-241705, SPL-240758	"File Integrity checks found 4281 files that did not match the system-provided manifest." shows in message but does not appear in the "Integrity Check of Installed Files" dashboard. Workaround: Clear the notification to dismiss the error and it will not appear again.
2023-06-30	SPL-241706, SPL-240758	"File Integrity checks found 4281 files that did not match the system-provided manifest." shows in message but does not appear in the "Integrity Check of Installed Files" dashboard. Workaround: Clear the notification to dismiss the error and it will not appear again.
2023-03-17	SPL-237480, SPL-243422, SPL-243423	CherryPy performance issues causing slow HTML document loading
2022-10-20	SPL-231830, SPL-239319, SPL-239320	SearchJob sometimes fails and returns error "Search <ID> not found. The search may have been cancelled while there are still subscribers" Workaround: Remark : this Splunk Enterprise issue may impact ITSI UI with loading issues (KPI and thresholds preview, Share Base search validation, Entity import, Maintenance windows preview) Fix : upgrade Splunk to a fixed version
2022-05-31	SPL-225037	Remote dataset dropdown menu resets to "Index" after selecting federated provider
2021-12-21	SPL-216787	Searches are cancelled or time out when the user leaves the browser window or switches tabs. Workaround: Change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62. Details This issue is caused by power saving settings in recent browser versions, where Javascript timers may be throttled. The user typically sees the following message in the search window on foreground searches: DAG Execution Exception: Search has been cancelled Search auto-canceled The search job has failed due to an error. You may be able to view the job in the Job Inspector
2017-07-13	SPL-143111	"Splunkd daemon is not responding" when edit local windows event log collection

Windows-specific issues

Date filed	Issue number	Description
2024-11-08	SPL-265859, SPL-265863, SPL-265864, SPL-265865, SPL-265866	A missing CloseHandle() can lead to memory leaks Workaround: Disable introspection
2024-09-05	SPL-262273, SPL-262271	Fix perfmon counter capped at 100
2024-09-05	SPL-262275, SPL-262271	Fix perfmon counter capped at 100
2024-09-05	SPL-262274, SPL-262271	Fix perfmon counter capped at 100
2024-07-18	SPL-259217, SPL-265734, SPL-265735, SPL-265736, SPL-265737	Denylist is not working (blacklist1 = EventCode="4662" Message="Account Name:(?!\s*admin1)")
2022-03-19	SPL-221019	WEC + subscription with ContentFormat "Events" - indexed ForwardedEvents show "Splunk could not get the description for this event" for the "Message" field Workaround: Following steps should be followed: - to reconfigure subscription type to RenderedText: wecutil ss <subscription-name> /cf:RenderedText - in order to work around a MS defect on the WindowsEventViewer causing field description resolution failures within the WindowsEventViewer, when configuring RenderedText contentFormat you might want to also change the subscription locale, if not already done, to en-US: wecutil ss <subscription-name> /l:en-US and the same also for the datetime format on the WEC server to English (United States), see also here: https://serverfault.com/questions/606144/win2012r2-eventlog-subscription-dont-display-informations https://social.technet.microsoft.com/Forums/ie/en-US/3fd3d1fc-1194-4899-978c-3283085648bc/eventlog-forwarding-issues-either-the-component-that-raises-this-event-is-not-installed-on-your - please make sure to install the most recent Windows add-on compatible with your Splunk release, following the official installation documentation: https://docs.splunk.com/Documentation/AddOns/released/Windows/Install - please configure inputs.conf on the splunk instance running on the WEC server as follows, in order to onboard the ForwardedEvents data in XML format: [WinEventLog://ForwardedEvents] renderXml = true then save and restart splunk in order to apply the changes. - last, but not least, unless renderXml was set to true already before installing/upgrading to a regressed version, you will need to rewrite your searches and reports in order to comply with the new/XML-specific field extractions shipped in the Windows add-on, since the data is now onboarded in XML format.

REST, Simple XML, and Advanced XML issues

Date filed	Issue number	Description
2020-07-28	SPL-192792	tsidxWritingLevel and other fields are set empty after updating index in UI
2017-07-13	SPL-143111	"Splunkd daemon is not responding" when edit local windows event log collection
2016-10-31	SPL-131072	Datamodel backend allows invalid time values

PDF issues

Date filed	Issue number	Description
2016-11-23	SPL-132925	Table data rows generated with the addcoltotals command do not show up in PDF Workaround: If you are using addcoltotals to generate a totals data row, renaming the _time field can cause PDF generation issues. Remove the label and labelfield or change the label to a number to generate the PDF as expected.

Admin and CLI issues

Date filed	Issue number	Description
2024-04-26	SPL-254998	effective concurrency limit for scheduled searches in not updating in search prefs manager page
2023-04-03	SPL-238114	messages.conf roles attribute not working as documented in messages.conf.spec
2021-03-26	SPL-203060	The splunkd process changes the local distsearch.conf on service start Workaround: There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will: Remove any settings that define default values already set in the /default/distsearch.conf file. Removes comments preceded by a hash. Reorders the KV pairs alphanumerically within a stanza. Reorders stanzas within the file.
2020-07-28	SPL-192792	tsidxWritingLevel and other fields are set empty after updating index in UI
2020-04-14	SPL-186365	Users are able to create/clone knowledge objects into apps where they lack permissions
2019-08-05	SPL-174406, SPL-109254	Root unable to run splunk cli if SPLUNK_OS_USER is set
2018-08-13	SPL-158658	A timeout or slow response when accessing Splunk Web Licensing page Workaround: A timeout or slow performance of the license management page is caused by a build-up of historical license warning messages, which are processed every time the page is accessed. Can be verified by running this search on the License Manager: | rest splunk_server=local /services/licenser/messages If a high value is returned for that end point, you are likely affected. Log a support ticket with Splunk to obtain a license reset key, and apply the key to clear out any historical license warning messages. After the reset license is applied, the license management pages should load normally.
2017-11-29	SPL-146820	Unable to access some settings/manager pages (data model editor) if starting from the setup page of a non-visible app Workaround: Navigate to a visible app, such as the search and reporting app, and access the Splunk settings pages from that app context.
2017-04-03	SPL-140747	SSL connection in Python when using new ciphers may be slow.
2016-11-09	SPL-131880	Reports/Alerts owned by the deleted user cannot be found in the Orphaned filter for the Reassign Knowledge Objects page

Uncategorized issues

Date filed	Issue number	Description
2024-11-20	SPL-266508, SPL-268398, SPL-268399, SPL-268400, SPL-268401	Fix Security Vulnerabilities in Hunk
2024-07-19	SPL-259311, SPL-263863, SPL-263864, SPL-263865, SPL-263866	Delayed creation of knowledge bundle
2024-05-24	SPL-256405, SPL-265289, SPL-265291, SPL-265292, SPL-265293, SPL-265294, SPL-265295, SPL-265296, SPL-265297, SPL-265290	HTTPServer does not read allowEmbedTokenAuth after certificate rotation / web server reload
2024-03-13	SPL-252573, SPL-251434	Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment [7]
2024-03-13	SPL-252571, SPL-251434	Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment [8]
2024-03-13	SPL-252572, SPL-251434	Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment [9]
2023-12-10	SPL-248188, SPL-248140	Slow indexer detection calculate send queue bytes
2023-12-10	SPL-248187, SPL-248140	Slow indexer detection calculate send queue bytes
2023-11-07	SPL-246765, SPL-245974	HTTP Event Collector s2s endpoint ignores all inputs.conf.spec.
2023-11-07	SPL-246766, SPL-245974	HTTP Event Collector s2s endpoint ignores all inputs.conf.spec.
2023-11-03	SPL-246640	web.conf server.socket_host no longer overrides splunk-launch.conf SPLUNK_BINDIP Workaround: No workaround available.
2023-09-25	SPL-245071, SCP-64986	Splunk Assist causes excessive logging before activation, sometimes on instances that do not run Splunk Assist at all Workaround: Disable Splunk Assist fully on those instances, see "Turn off Splunk Assist" in the Splunk Documentation for the procedure: https://docs.splunk.com/Documentation/Splunk/9.1.1/DMC/ActivateAssist#Turn_off_Splunk_Assist Modular inputs can be disabled individually in $SPLUNK_HOME/etc/apps/splunk_assist/local/inputs.conf
2023-05-07	SPL-239645, SPL-267229	Cascading bundle replication stops due to an outage of an indexer and SH Captain does not generate new KO bundles till restarted or captaincy is transferred to another SH member Workaround: restart captain or transfer captaincy to another SHC member
2023-05-02	SPL-240700	A forwarder that does not have a 'clientCert' set in its outputs.conf configuration file will not connect over TLS to a receiver even when the receiver has 'requireClientCert' set to "false" in its inputs.conf configuration file Workaround: Specify a value for 'clientCert' in the outputs.conf file on the forwarder. Ensure that the file you specify for the setting exists and is a valid file in Privacy Enhanced Mail (PEM) format.
2023-01-06	SPL-234643	Splunkd abort - due to 3rd party S2S client unable to process ACKs. Workaround: For some versions of 3rd-party S2S client, there is an option to change the behavior of a failed ACK. For example, turning off "Minimize in-flight data loss".
2022-11-14	SPL-232803	Job endpoint /services/search/jobs not returning QUEUED jobs Workaround: Queued job displays using job endpoint with SID: | rest /services/search/jobs/1668102339.174_23558BC9-6A39-4F4A-9FD2-968C358489B7 splunk_server=local
2022-05-11	SPL-224063, SPL-258953	metrics.log - tcpin_connections - logs are merging from different forwarders in single events
2022-04-06	SPL-222105	When all inherited roles are taken out from admin role, it will cause admin user failed to show other users even though all capabilities is set natively. Workaround: Two possible approaches: 1. Remove the option grantableRoles = admin from authorize.conf - this is not permanent workaround and will need to be done every time admin role is modified. 2. Add any capabilities that the other user roles have to the "admin" role.
2021-04-24	SPL-204740, SPL-204735	Deletion of a workload pool is allowed if there is a 'disabled' rule that is related to that workload pool and this can cause errors if the rule is re-enabled later Workaround: To prevent this issue: When you delete a workload pool, please make sure that you delete any disabled workload rules that are associated with that workload pool. To resolve the issue if you encounter this: Disable or delete the workload rule that is associated with a workload pool that does not exist anymore.
2021-03-19	SPL-202682	The license usage report tab name is Previous 60 days, but the reports run over the last 30 days
2021-02-10	SPL-200532	SmartStore: Stuck fixup due to inability to freeze unsearchable/unstable bucket Workaround: This issue is caused by a single unsearchable bucket that has been frozen while not existing on remote storage. The bucket copy on the peer node's cache remains stuck in the fixup state, resulting in messages to the effect that all data is not searchable, the replication factor is not met, and the search factor is not met. To resolve, on the peer node, invoke the "/services/cluster/peer/buckets" endpoint, specifying the faulty bucket, setting "search_state=Searchable" to make the bucket searchable. You do not need to restart the peer node afterwards. Here is the syntax for the required endpoint: curl -k -u admin https://<peer_node_with_bucket>:<mgmt_port>/services/cluster/peer/buckets/<bucket_id>/change_bucket -d bucket_mask=0 -d search_state=Searchable -d generation_id=0 -d searchable_sources="peer,site,server_name,host_port_pair,replication_port,replication_use_ssl,searchable,bucket_mask" Note that pairs of angle brackets indicate variables that must correspond to your instance and bucket.
2020-10-01	SPL-195810	Using CLI command to stop migration of KVstore on a SHC running on Windows OS can cause the SHC captain to reach an invalid state Workaround: Restart the SHC captain
2020-08-10	SPL-193389	Parallel upload is not supported in gcp-sse-kms encryption mode Workaround: In the volumes using gcp-sse-kms encryption mode, specify "remote.gs.upload_chunk_size = 0" to disable parallel upload.
2020-07-30	SPL-192936	Subsecond search - When you update metric.timestampResolution via the UI, it is not updated on the search head index.conf file. This does not affect search functionality.
2020-05-06	SPL-188800	Starting Splunk software with incorrect KV store storage engine causes KV store to crash Workaround: In the [kvstore] stanza of your server.conf file, set the storageEngine setting to match the storage engine that you're using, either wiredTiger or mmapv1. To learn which storage engine you're using, check whether the file extensions in the var/lib/splunk/kvstore/mongo directory are *.wt for Wired Tiger or *.ns for Memory Mapped.
2019-10-03	SPL-177447	Bundle replication takes longer than expected time for indexers that have bundleEnforcerBlacklist configured
2019-09-26	SPL-177144, SPL-177326	Under heavy search workload, the search memory usage estimation may be higher than actual usage
2019-09-25	SPL-177008, SPL-176710, SPL-177009	Workload management fails to enable for addition of a pool with 1% cpu and 1% memory
2019-09-16	SPL-176514	Offline rebuild of unsearchable bucket may lead to stale information in dbinspect searches
2019-09-13	SPL-176447	SmartStore: Migration uploads of auto_high_volume buckets can fail indefinitely due to an XFS bug Workaround: Before migration, lower the max_concurrent_uploads setting in server.conf to 2. After migration, revert the setting to the default of 8.
2019-07-19	SPL-173449, SPL-173259	timezone isn't stored for start_time/end_time of rule schedule every_day/every_week/every_month
2019-03-26	SPL-168314	SmartStore standalone instance + Monitoring Console: Bootstrapping panel needs to reflect the standalone bootstrapping process
2018-03-20	SPL-152330, SPL-151992	After installing Splunk on Windows using msiexec and the "GENRANDOMPASSWORD=1" option (and if generated password ends with backslash) admin is unable to login with msg "No users exist. Please set up a new user." Workaround: Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk [user_info] PASSWORD = <yourpassword>
2017-06-29	SPL-142789, SPL-95144	Indexed message for Windows security event logs shows "FormatMessage error" Workaround: Splunk believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service.
2017-05-09	SPL-141693	DataModel Editor - when child object has same name as inherited field, inherited field does not show in the inherited fields list.
2017-01-06	SPL-134707	Splunk restart does not create missing server.pem certificate on Windows Workaround: Use bin/splunk createssl server-cert -d etc/auth/ -n server to generate a new certificate.
2016-08-31	SPL-127800	Opting in to data sharing on a monitoring console produces duplicate data
2016-07-26	SPL-125052	Sole Admin can demote themself to Power without path of recovery in GUI. Workaround: Through the command line, you can open notepad and modify the password file to regain 'Admin' status.
2016-06-21	SPL-123174	JSON indexed_extractions doesn't work for TCP inputs

Splunk Analytics for Hadoop

Date filed	Issue number	Description
2017-04-04	ERP-2040	Splunk archiving fails for large block sizes (buckets) due to HDFS write crashes for Hadoop version 2.8, 2.7.x Workaround: Upgrade Hadoop to 2.8.2 or higher.
2015-09-09	ERP-1650	timestamp data type not properly deserialized.
2015-08-05	ERP-1619	Searching on a newly created archive index before the bucket copy saved search is run causes a filenotfound exception. Workaround: Reenable the bucket copy saved search and let it run, or force the archiving to happen via | archivebuckets force=1 and then rerun the search.
2015-07-07	ERP-1598	minsplit rampup - splits generation takes too long. Workaround: Set minsplits=maxsplits
2015-05-12	ERP-1502	Non-accelerated pivot search on Pivot UI page waits for a long time to return result.
2015-01-08	ERP-1343, SPL-95174	Splunk Analytics for Hadoop searches fail on corrupted journal.gz files, although Splunk searches run without error. Workaround: Add the journal.gz to the input path's blacklist (vix.input.1.ignore = ....)
2014-10-27	ERP-1216	Data Explorer preview does not honor existing sourcetypes for big5/sjis files.
2014-10-03	ERP-1164	Report acceleration summary gets deleted when two Splunk Analytics for Hadoop instances point to the same Splunk working directory. Workaround: To mitigate this issue, make sure that vix.splunk.home.hdfs (or Working directory in the UI) is unique on both search heads that are not in a pool. To keep your instances in the same working directory, configure vix.splunk.search.cache.path to be unique on both search heads.