Known issues
The following are issues and workarounds for this version of Splunk Enterprise. Splunk Enterprise 9.2.0 was released on January 31, 2024. Splunk Enterprise 9.2.0.1 was released on February 8, 2024 to correct a non-security issue that can affect cluster managers during bundle pushes. Splunk recommends that customers use version 9.2.0.1 instead of version 9.2.0.
Issues are listed in all relevant sections. Some issues appear more than once.
Refer to System requirements in the Installation Manual for a list of supported platforms and architectures.
For a list of deprecated features and platforms, refer to Deprecated features and removed features in this manual.
Highlighted issues
Date filed | Issue number | Description |
---|---|---|
2024-02-05 | SPL-250501 | Config validation check mistakenly blocks config push on Splunk Enterprise. This issue affects Splunk Enterprise 9.2.0 and is fixed in Splunk Enterprise 9.2.0.1. |
2024-02-05 | SPL-250529 | Occasional indexer crashes during data ingest. This issue affects Splunk Enterprise 9.2.0 and is fixed in Splunk Enterprise 9.2.0.1. |
Upgrade issues
Date filed | Issue number | Description |
---|---|---|
2024-04-04 | SPL-253690, SPL-247255 | Issue with Splunk Enterprise version 9.1.x and 9.2.0 when connected to proxy server Workaround: An enhancement was introduced to the search process in Splunk Enterprise version 9.1.1 that optimizes searches by using the peer's IP address instead of querying DNS for the target peers. If an http_proxy is specified in the server.conf file, the enhancement causes the originating peer's IP address to fail to resolve. As a result, the IP address of the proxy, instead of the originating peer, is logged and utilized in the peers.csv file during search operations, causing the following error message to display in Splunk Web: "Received error from proxy server".
[distributedSearch] useIPAddrAsHost=false |
2024-02-20 | SPL-251301 | Unable to install/upgrade Splunk Enterprise & UF RPM package v9.2.x on the same server. |
2020-08-31 | SPL-194426 | External search command chunked v2 python SDK fails with multibyte result data under python 3. Workaround: Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters. App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps. Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available. |
2020-07-10 | SPL-191850 | The .deb installation package will fail if dpkg version doesn't support an .xz compressed control file. Workaround: Update dpkg to version 1.17.6 or later. |
2018-04-13 | SPL-153403 | After running the "clean userdata" command, admin is unable to login with msg "No users exist. Please set up a new user." Workaround: Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk [user_info] |
Data input issues
Date filed | Issue number | Description |
---|---|---|
2024-01-13 | SPL-249543, SPL-251748, SPL-251749, SPL-253929, SPL-251746, SPL-253927, SPL-253928 | TcpInputProcessor not able to drain splunktcpin queue during graceful shutdown. Workaround: Splunk recommends customers set `useACK` to true to ensure in-memory is not dropped in the event of indexer rolling restarts or repaving. Thus, the best short-term solution is to set `useACK` to `true`. |
2024-01-10 | SPL-249424, SPL-249409 | Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x |
2024-01-10 | SPL-249423, SPL-249409 | Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x |
2024-01-10 | SPL-249422, SPL-249409 | Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x |
2023-11-07 | SPL-246769, SPL-243845 | HTTP Input HEC input ignores _meta in inputs.conf |
2023-11-07 | SPL-246768, SPL-243845 | HTTP Input HEC input ignores _meta in inputs.conf |
2023-06-15 | SPL-241076, SPL-251249, SPL-251251, SPL-251329, SPL-251250 | Metrics event can be indexed in default event index when mcollect is used. Workaround: Avoid to restart if queue is blocked. (wish maybe should be best practise but the revert is what always happen customer queue blocked let restart to solve it .) |
2022-08-17 | SPL-228646, SPL-228645 | Restart is needed when AWS access key pairs rotate (w/o grace period) or other S3 config settings for Ingest Actions become invalid |
2022-08-09 | SPL-228117, SPL-257140 | "file" is incorrectly listed as a supported scheme for ingest actions in outputs.conf.spec |
2022-04-08 | SPL-222366 | Ingest Actions does not work with Splunk Free, Personalized Devtest, Developer, and Forwarder-only licenses |
Search issues
Date filed | Issue number | Description |
---|---|---|
2024-09-06 | SPL-262305, SPL-253147 | Crashing thread: TcpOutEloop Workaround: https://community.splunk.com/t5/Knowledge-Management/Splunk-crash-during-tcpout-outputs-conf-reload/td-p/699397 |
2024-05-09 | SPL-255514 | "| timechart count" search is causing Splunk to crash with "Crashing thread: searchOrchestrator" Workaround: When using a search with only the timechart command in a search: | timechart count Splunk instance will crash with "Crashing thread: searchOrchestrator". Currently there is no workaround other than not using this search string |
2024-04-12 | SPL-254077, SPL-241370 | CIDR match for tstats with ipv6 addresses isn't supported Workaround: The tstats command currently doesn't filter events with CIDR match on fields that contain IPv6 addresses. Running tstats searches containing IPv6 addresses might result in the following error indicating that the addresses are treated as non-exact queries:
Error in 'TsidxStats': WHERE clause is not an exact query |
2024-04-04 | SPL-253690, SPL-247255 | Issue with Splunk Enterprise version 9.1.x and 9.2.0 when connected to proxy server Workaround: An enhancement was introduced to the search process in Splunk Enterprise version 9.1.1 that optimizes searches by using the peer's IP address instead of querying DNS for the target peers. If an http_proxy is specified in the server.conf file, the enhancement causes the originating peer's IP address to fail to resolve. As a result, the IP address of the proxy, instead of the originating peer, is logged and utilized in the peers.csv file during search operations, causing the following error message to display in Splunk Web: "Received error from proxy server".
[distributedSearch] useIPAddrAsHost=false |
2023-12-18 | SPL-248552 | ProcessDispatchedSearch error displayed - The process cannot access the file because it is being used by another process Workaround: There is no workaround. Details The only impact of this error is that some log lines might be dropped from Splunk's own search.log and splunkd.log files. Searches still work as usual. |
2023-12-12 | SPL-248297 | Higher memory usage than usual on Windows systems after upgrade from Splunk Enterprise version 9.0/8.x to version 9.1.x/9.2.x Workaround: As of Splunk Enterprise version 9.1.x, a new system call for allocating memory called mimalloc has been implemented on Windows. Prior to Splunk Enterprise version 9.1, the Splunk Enterprise implementation for Windows used a default system call. The new mimalloc system call can cause an increase in memory usage, which impacts search heads, indexers, and universal forwarders, especially when data model acceleration is in use. This issue will be fixed in Splunk Enterprise versions 9.1.5 and 9.2.2. Upgrade to Splunk Enterprise version 9.1.5 or 9.2.2 when they become available. |
2023-06-09 | SPL-240774 | The DELIMS setting or the kvdelim option may not be applied correctly when the k/v delim character appears 2 or more times in a field value Workaround: Perform field extractions by modifying your searches using other commands, such as the rex command or eval command. |
2023-04-14 | SPL-238738 | Federated Search for Splunk does not support the "Show Source" Field Action |
2023-03-28 | SPL-237902 | Ad hoc searches that specify earliest relative time offset assuming from 'now' should explicitly include 'latest=now' to avoid a potential time range inaccuracy Workaround: Ad hoc searches searches that use the earliest time modifier with a relative time offset should also include 'latest=now' in order to avoid time range inaccuracies. For example, if you want to get all events from the last 10 seconds starting at 01:00:10, the following search returns all events that occur between the time of 01:00:00 and 01:00:10, as expected. index=main earliest=-10s latest=now Running the same search without including 'latest=now' might produce unpredictable results or impact performance in certain scenarios when the search head is overloaded with ad hoc searches. See Specify earliest relative time offset and latest time in ad hoc searches in the Splunk platform Search Manual. |
2022-07-29 | SPL-227633 | Error : Script execution failed for external search command 'runshellscript' Workaround: The setting precalculate_required_fields_for_alerts=0 can be set on saved searches that have no other alert actions attached aside from the "Run A Script" action, to quash the error. For saved searches that have multiple alert action attached, this may not be safe as it will disable back propagation of required fields for all alert actions, which might result in the parent search extracting more fields than required, which could negatively impact performance for that search. |
2021-12-21 | SPL-216787 | Searches are cancelled or time out when the user leaves the browser window or switches tabs. Workaround: In Splunk Enterprise 8.1.7, 8.2.4, and higher change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62.Details
|
2021-09-22 | SPL-212495, SPL-196040, SPL-219811 | Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles Workaround: none |
2020-12-04 | SPL-198284, SPL-231587 | Crash in search process in PrecacheUsersThread when max_searches_per_process is set lower than default Workaround: Set limits.conf back to default, by removing any override of max_searches_per_process. For example: [search] max_searches_per_process=1 to [search] |
2020-08-31 | SPL-194426 | External search command chunked v2 python SDK fails with multibyte result data under python 3. Workaround: Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters. App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps. Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available. |
2020-02-12 | SPL-183259 | When generating LISPY for field values that are numbers (""), the values aren't deduplicated, which can cause slowdowns in certain scenarios Workaround: Dedup values in search before, for example: instead of index="field_test" [search index="field_test" globalCallID_callId=1234* | fields globalCallID_callId] add a stats or dedup in the subsearch: index="field_test" [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ] If that list is still large and you're seeing the slowdown, consider moving the filtering to a | where after the initial search, for example: index="field_test" globalCallID_callId=* | where [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ] |
2020-01-10 | SPL-181573 | geostats provides incorrect results for lower zoom levels when split BY has a higher cardinality than globallimit. Workaround: - Increase globallimit to the value of "unique values" number mentioned in the warning message: "The split by field <field> has a large number of unique values <number>. Chart column set will be trimmed to 10. Use globallimit argument to control column count." - Use very high globallimit in geostats and post process after if needed - Don't use BY in geostats - Use lower cardinality BY and/or higher globallimit in geostats |
2017-07-13 | SPL-143111 | "Splunkd daemon is not responding" when edit local windows event log collection |
2017-04-04 | SPL-140765 | Splunk having problems extracting json file consisting of 68k plus key-value pairs |
2016-11-29 | SPL-133182 | When two datasets have identical names but one is local (private) while the other is global, attempts to view or extend the global dataset use results from the local dataset instead. |
2014-10-02 | SPL-91638, SPL-107375 | For scheduled searches in a search head cluster, empty search jobs may appear in the job inspector for a cluster member. |
Federated search issues
Date filed | Issue number | Description |
---|---|---|
2024-10-15 | SPL-264529, SPL-261801 | Search with mcatalog command returns missing metrics when used with append=t and last index is not valid |
2024-05-23 | SPL-256393, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
2024-05-23 | SPL-256394, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
2024-05-23 | SPL-256392, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
2024-05-23 | SPL-256391, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
2024-05-23 | SPL-256390, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
2024-04-23 | SPL-254718, SPL-253248, SPL-255069 | Federated searches not completing with error "Socket error during transaction. Socket error: Success" |
2024-04-19 | SPL-254540, SPL-253986 | Transparent Federated Search should not ignore federated service account index permission when fsh user is set to SPLUNK_SYSTEM_USER |
2024-04-19 | SPL-254539, SPL-253986 | Transparent Federated Search should not ignore federated service account index permission when fsh user is set to SPLUNK_SYSTEM_USER |
2024-04-05 | SPL-253755, SPL-252488, SPL-253757 | federated search should alert ( and block the search ) when it is run in realtime mode |
2024-03-27 | SPL-253248, SPL-254718, SPL-254719, SPL-254720, SPL-254722, SPL-254721 | Federated searches not completing with error "Socket error during transaction. Socket error: Success" |
2024-03-12 | SPL-252488, SPL-248786, SPL-253755 | Lookups in transparent mode don't use proper lookup when fsh and rsh have lookup with same name |
2024-03-11 | SPL-252400 | federated search for Splunk: a wildcard search on a search head that has unreachable federated indexes should not fail Workaround: Make sure that standard federatd provider is reachable |
2024-01-25 | SPL-250070 | Bugfix for remoteTLCmd duplicated during phase generation in Verbose Mode |
2024-01-18 | SPL-249666, SPL-244551 | FS-StandardMode : Standalone sub-search with HEAD doesn't return any results |
2024-01-10 | SPL-249387, SPL-250067, SPL-250069, SPL-250567 | Bugfix for remoteTLCmd duplicated during phase generation in Verbose Mode |
2023-12-21 | SPL-248786, SPL-252486, SPL-252487, SPL-252488 | Lookups in transparent mode don't use proper lookup when fsh and rsh have lookup with same name Workaround: If the `rsh` is getting transparent searches and it does not have indexers connected to it, the `rsh` does not look on the kvstore values that were sent to it from the `fsh`. The workaround is to have indexers attached to the `rsh` |
2023-10-31 | SPL-246556, SPL-249728, SPL-249746 | Federated searches that contain stats count by a field that doesn't exist return 0 events when run in verbose mode |
2023-09-20 | SPL-244927, SPL-244124, SPL-247063, SPL-246460, PSRT-7170 | Federated searches that include 'table' or 'rex' commands return 0 events when run in verbose mode |
2023-09-05 | SPL-244248, SPL-239298 | Federated Search, Enterprise --> Cloud configuration: Performance degradation increases when the number of indexers increases in the RSH Workaround: One possible workaround is to use a more efficient query. For example, use "| tstats count where index=main by splunk_server" instead of "index=main | stats count by splunk_server".
Use this workaround if you are using your Splunk Enterprise federated search head (FSH) instance only for running federated searches. This workaround might affect non-federated searches. On the Splunk Enterprise FSH, follow these steps:
|
2023-07-20 | SPL-242282, SPL-242864 | Federated Searches fail for union commands when query optimization diverge between FSH x RSH |
2023-05-02 | SPL-239436 | In standard mode federated search, outputlookup existence check on RSH causes search to terminate early although it is not run on RSH Workaround: Define the lookup on both federated search head and remote search head. |
2023-04-14 | SPL-238738 | Federated Search for Splunk does not support the "Show Source" Field Action |
2023-04-10 | SPL-238501 | Federated search "outputlookup" command cannot add data to local lookup table Workaround: Define the same lookup on the remote search head too, so the remote search head will not error out early and return 0 results. |
2022-10-19 | SPL-231712 | Create/Edit Role - In the UI, the "Wildcards" tool cannot be used to specify allowed federated indexes for standard mode federated search |
2022-07-15 | SPL-226877 | Federated Search UI Error: Cannot create saved search dataset for federated index if dataset name contains space Workaround: Use REST API to create the federated saved search instead:
|
2022-05-31 | SPL-225037 | Remote dataset dropdown menu resets to "Index" after selecting federated provider |
2022-02-25 | SPL-219793 | Some commands in federated searches return incorrect resultCount values when run in verbose mode Workaround: Use Verbose and Smart mode specifically for searches with transforming commands like stats , chart , and timechart , and then review the results in the Statistics tab. To review event counts, run non-transforming searches in Fast mode.Search-time field extraction usually requires searches without transforming commands that run in either Verbose or Smart mode. When you run searches in Fast mode, you can ensure that search-time field extraction takes place for federated searches by appending |
2022-02-08 | SPL-218841 | Reporting command in verbose mode returns 0 events despite correct event_count |
2021-10-14 | SPL-213745, SPL-251131 | Standard mode federated search: Unable to set federated index as default index |
Saved search, alerting, scheduling, and job management issues
Date filed | Issue number | Description |
---|---|---|
2024-07-19 | SPL-259287, SPL-254139, SPL-262990 | Low Privileges user is able to modify dispatchAs field from Owner to User |
2024-07-19 | SPL-259288, SPL-254139, SPL-262991 | Low Privileges user is able to modify dispatchAs field from Owner to User |
2024-07-19 | SPL-259283, SPL-254139, SPL-262989 | Low Privileges user is able to modify dispatchAs field from Owner to User |
2023-11-08 | SPL-246785, SPL-244383 | Search-Scheduler Splunk Crashes on Job Servers in SHC. Workaround: Workaround collections.conf [LoggedOutSessionTokens] disabled = true server.conf
[general]
invalidateSessionTokensOnLogout = false |
2023-07-21 | SPL-242301, SPL-231558 | The UI trigger for summary rebuild doesn't work for some accelerated data models that have no root-event dataset and have a reporting command in first root search dataset Workaround: The workaround is to change the Data Model definition to reorder the root search objects such that the root search object that can be accelerated is the very first one in the list. For instance, for the provided `test_internal_audit_logs.json`, edit the JSON file on disk and move `failed_searches` dataset before `fully_completed_searches`. |
2023-07-07 | SPL-241821 | Data Model Accelerations that have Automatic Rebuilds enabled may lead to unbounded memory growth due to search expansion, resulting in Out of Memory errors Workaround: For a data model that is experiencing high memory usage, perform the following steps:
Furthermore, applying index constraints to restrict the list of indexes searched for building a given DMA summary and applying tags allowlisting would help curtail the memory usage. |
2019-09-20 | SPL-176812 | Multiple SH Clustering with single deployer can't use datamodel summary sharing |
2018-09-19 | SPL-160286 | The data preview for the Add Data workflow does not display for Log to Metrics source types |
2017-11-29 | SPL-146802 | Distributed environment requires index defined on search head for log event alerts |
2017-08-14 | SPL-143947 | Report acceleration is broken for users with a configured role-based access filter |
Charting, reporting, and visualization issues
Date filed | Issue number | Description |
---|---|---|
2024-01-05 | SPL-249098, SPL-247587 | After upgrading to 9.1.1 dashboards iframe panels load to a blank page |
2023-11-23 | SPL-247466 | Dashboard Studio layers button is not working in Windows 10 Workaround: Change object layers by manually editing the order of elements in the layout structure in the dashboard definition. |
2023-09-18 | SPL-244788, SPL-247096, SPL-247097 | "Awaiting user confirmation" error when img src is a token that is set to a URL after SXML dashboard loads Workaround: If possible, hard code the img src domain and capture the rest of the img URL as a token. For example, instead of `<img src="$image_url$">`, use `<img src="https://www.example.com/$image_url$">` and remove the domain from the value of $image_url$. Another option is to add a hidden placeholder image to the HTML panel with the issue: <style> .placeholder-img { display:none } </style>
<img class="placeholder-img" src="https://dev.splunk.com/test.jpg" /> |
2023-06-14 | SPL-240965 | Dashboard Studio home dashboard flickers on specific viewport resolutions with scrollbar visibility set to always Workaround: Resize the viewport or current monitor resolution. If the issue persists, try changing the dimensions of the dashboard. |
2023-06-08 | SPL-240750 | Inconsistency in displayed timezone in Dashboard Studio when using time range tokens |
2016-04-27 | SPL-118911 | In SHC, referenced saved real-time searches in a dashboard do not stream results. Workaround: See Troubleshoot referenced real-time searches for workaround details.
|
Distributed search and search head clustering issues
Date filed | Issue number | Description |
---|---|---|
2021-09-22 | SPL-212495, SPL-196040, SPL-219811 | Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles Workaround: none |
2021-03-26 | SPL-203060 | The splunkd process changes the local distsearch.conf on service start Workaround: There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:
|
2017-11-29 | SPL-146802 | Distributed environment requires index defined on search head for log event alerts |
2017-03-13 | SPL-138654 | Splunk searches fail when filepath gets too long on Windows |
2016-07-12 | SPL-124085 | On Search Head Cluster It is not possible to remove an App from the SHs once it has been disabled. |
Data model and pivot issues
Date filed | Issue number | Description |
---|---|---|
2023-07-21 | SPL-242301, SPL-231558 | The UI trigger for summary rebuild doesn't work for some accelerated data models that have no root-event dataset and have a reporting command in first root search dataset Workaround: The workaround is to change the Data Model definition to reorder the root search objects such that the root search object that can be accelerated is the very first one in the list. For instance, for the provided `test_internal_audit_logs.json`, edit the JSON file on disk and move `failed_searches` dataset before `fully_completed_searches`. |
2023-07-07 | SPL-241821 | Data Model Accelerations that have Automatic Rebuilds enabled may lead to unbounded memory growth due to search expansion, resulting in Out of Memory errors Workaround: For a data model that is experiencing high memory usage, perform the following steps:
Furthermore, applying index constraints to restrict the list of indexes searched for building a given DMA summary and applying tags allowlisting would help curtail the memory usage. |
2019-09-20 | SPL-176812 | Multiple SH Clustering with single deployer can't use datamodel summary sharing |
Indexer and indexer clustering issues
Date filed | Issue number | Description |
---|---|---|
2024-05-29 | SPL-256658, SPL-255517 | Indexer Discovery deadlock during tcpout reload |
2024-05-29 | SPL-256657, SPL-255517 | Indexer Discovery deadlock during tcpout reload |
2024-05-27 | SPL-256435, SPL-256073 | _internal index data not searchable -- a corruption fixup is in progress or has failed for a bucket |
2024-04-04 | SPL-253649, SPL-246435 | Rolling Restart generate fix-up task that search factor taking hours complete as search files replication fails. |
2024-01-13 | SPL-249543, SPL-251748, SPL-251749, SPL-253929, SPL-251746, SPL-253927, SPL-253928 | TcpInputProcessor not able to drain splunktcpin queue during graceful shutdown. Workaround: Splunk recommends customers set `useACK` to true to ensure in-memory is not dropped in the event of indexer rolling restarts or repaving. Thus, the best short-term solution is to set `useACK` to `true`. |
2016-08-25 | SPL-127353 | Data rebalance finishes early when one peer is the source for all buckets Workaround: when only one indexer in a cluster indexed data (has all the searchable copies), rebalance once before adding the new indexer, and then rebalance a second time |
Universal forwarder issues
Date filed | Issue number | Description |
---|---|---|
2024-02-22 | SPL-251517, SPL-237849 | CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 Upgrade to version 9.0.0.1 issue doesn't appear. [2]
The detailed workaround didn't solve the issue. |
2024-02-22 | SPL-251515, SPL-237849 | CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 Upgrade to version 9.0.0.1 issue doesn't appear. [4]
The detailed workaround didn't solve the issue. |
2024-02-22 | SPL-251516, SPL-237849 | CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 Upgrade to version 9.0.0.1 issue doesn't appear. [6]
The detailed workaround didn't solve the issue. |
2024-02-20 | SPL-251301 | Unable to install/upgrade Splunk Enterprise & UF RPM package v9.2.x on the same server. |
2023-12-19 | SPL-248587, SPL-252796, SPL-252797 | Unable to install or upgrade to Splunk Universal Forwarder version 9.1.3 Workaround: Install UF while passing the following feature flag: msiexec.exe /i $SPLUNK_MSI_PACKAGE USE_LOCAL_SYSTEM=1 |
2022-08-17 | SPL-228646, SPL-228645 | Restart is needed when AWS access key pairs rotate (w/o grace period) or other S3 config settings for Ingest Actions become invalid |
2022-06-23 | SPL-226019 | Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality. |
2022-03-23 | SPL-221239 | System Introspect App fails when universal forwarder is installed at non-admin user |
Distributed deployment, forwarder, deployment server issues
Date filed | Issue number | Description |
---|---|---|
2024-03-18 | SPL-252818, SPL-253411, SPL-254631 | Deployment Server not displaying Apps correctly after 9.2.0.1 update Workaround: Deployment Clients are showing apps as not having been deployed and Server Classes do not show their matching clients. This is just a display issue, the DS will continue to honor the server class filters when deploying the apps. As a workaround, use the server class's preview function to list the deployment clients. Alternatively, do not use machine type to filter your deployment clients. |
Monitoring Console issues
Date filed | Issue number | Description |
---|---|---|
2024-06-28 | SPL-258394 | Health Report for destination output issues show Last 50 detailed logs in Indexer Cluster nodes but not in Search Head or Cluster Manager |
2023-12-08 | SPL-248160, SPL-243155 | The Monitoring Console does not show Indexer and Search Head statistics on the Overview page |
2021-03-29 | SPL-203100 | Summary page on monitoring console doesn't show correct RF/SF when not running on the CM. |
2019-11-13 | SPL-179528 | The splunktcp and splunktcp-ssl stanzas are not reloadable in inputs.conf |
2017-08-14 | SPL-143981 | Uninstall app dialog does not show the app name correctly when the app doesn't have the label |
2017-05-24 | SPL-141982 | Upload modal should use size=large File element |
2017-04-19 | SPL-141274 | Clicking Install multiple times in Install dialog causes error |
2016-11-14 | SPL-132151 | XML error when trying to download uninstalled app |
Splunk Web and interface issues
Date filed | Issue number | Description |
---|---|---|
2024-06-04 | SPL-256902 | Splunk is crashing with "Crashing thread: WebuiStartup" when TLS is used for Splunk Web with an empty DNS under "X509v3 Subject Alternative Name" field of the certificate Workaround: Remove empty DNS entires, for example "DNS:" or "DNS", from "X509v3 Subject Alternative Name" field of the certificate used in "serverCert" settings under [settings] stanza of web.conf. |
2022-05-31 | SPL-225037 | Remote dataset dropdown menu resets to "Index" after selecting federated provider |
2021-12-21 | SPL-216787 | Searches are cancelled or time out when the user leaves the browser window or switches tabs. Workaround: In Splunk Enterprise 8.1.7, 8.2.4, and higher change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62.Details
|
2017-07-13 | SPL-143111 | "Splunkd daemon is not responding" when edit local windows event log collection |
Windows-specific issues
Date filed | Issue number | Description |
---|---|---|
2024-09-05 | SPL-262273, SPL-262271 | Fix perfmon counter capped at 100 |
2024-09-05 | SPL-262275, SPL-262271 | Fix perfmon counter capped at 100 |
2024-09-05 | SPL-262274, SPL-262271 | Fix perfmon counter capped at 100 |
2024-09-04 | SPL-262047, SPL-257961 | On Windows Splunk Enterprise Platform process instrument-resource-usage continuing growing its handles count for handle object Process. Workaround: Either proactively monitor and restart Splunk or kill splunkd process "instrument-resource-usage". Alternatively, disable introspection altogether. In the introspection_generator_addon app add the [introspection:generator:resource_usage] stanza in %SPLUNK_HOME%\etc\apps\introspection_generator_addon\local\server.conf
as follow:
[introspection:generator:resource_usage]
disabled = true
acquireExtra_i_data = false |
2024-09-04 | SPL-262045, SPL-257961 | On Windows Splunk Enterprise Platform process instrument-resource-usage continuing growing its handles count for handle object Process. Workaround: Either proactively monitor and restart Splunk or kill splunkd process "instrument-resource-usage". Alternatively, disable introspection altogether. In the introspection_generator_addon app add the [introspection:generator:resource_usage] stanza in %SPLUNK_HOME%\etc\apps\introspection_generator_addon\local\server.conf
as follow:
[introspection:generator:resource_usage]
disabled = true
acquireExtra_i_data = false |
2024-09-04 | SPL-262049, SPL-257961 | On Windows Splunk Enterprise Platform process instrument-resource-usage continuing growing its handles count for handle object Process. Workaround: Either proactively monitor and restart Splunk or kill splunkd process "instrument-resource-usage". Alternatively, disable introspection altogether. In the introspection_generator_addon app add the [introspection:generator:resource_usage] stanza in %SPLUNK_HOME%\etc\apps\introspection_generator_addon\local\server.conf
as follow:
[introspection:generator:resource_usage]
disabled = true
acquireExtra_i_data = false |
REST, Simple XML, and Advanced XML issues
Date filed | Issue number | Description |
---|---|---|
2020-07-28 | SPL-192792 | tsidxWritingLevel and other fields are set empty after updating index in UI |
2017-07-13 | SPL-143111 | "Splunkd daemon is not responding" when edit local windows event log collection |
PDF issues
Date filed | Issue number | Description |
---|---|---|
2016-11-23 | SPL-132925 | Table data rows generated with the addcoltotals command do not show up in PDF Workaround: If you are using addcoltotals to generate a totals data row, renaming the _time field can cause PDF generation issues.
Remove the label and |
Admin and CLI issues
Date filed | Issue number | Description |
---|---|---|
2024-04-26 | SPL-254998 | effective concurrency limit for scheduled searches in not updating in search prefs manager page |
2021-03-26 | SPL-203060 | The splunkd process changes the local distsearch.conf on service start Workaround: There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:
|
2020-07-28 | SPL-192792 | tsidxWritingLevel and other fields are set empty after updating index in UI |
2020-04-14 | SPL-186365 | Users are able to create/clone knowledge objects into apps where they lack permissions |
2019-08-05 | SPL-174406, SPL-109254 | Root unable to run splunk cli if SPLUNK_OS_USER is set |
2018-08-13 | SPL-158658 | A timeout or slow response when accessing Splunk Web Licensing page Workaround: A timeout or slow performance of the license management page is caused by a build-up of historical license warning messages, which are processed every time the page is accessed. Can be verified by running this search on the License Manager: | rest splunk_server=local /services/licenser/messages If a high value is returned for that end point, you are likely affected. Log a support ticket with Splunk to obtain a license reset key, and apply the key to clear out any historical license warning messages. After the reset license is applied, the license management pages should load normally. |
2017-11-29 | SPL-146820 | Unable to access some settings/manager pages (data model editor) if starting from the setup page of a non-visible app Workaround: Navigate to a visible app, such as the search and reporting app, and access the Splunk settings pages from that app context. |
2017-11-07 | SPL-146255 | limits.conf enable_clipping cloropleth setting is app/user tunable rather than global like the rest of limits.conf |
2017-04-03 | SPL-140747 | SSL connection in Python when using new ciphers may be slow. |
2016-11-09 | SPL-131880 | Reports/Alerts owned by the deleted user cannot be found in the Orphaned filter for the Reassign Knowledge Objects page |
Uncategorized issues
Date filed | Issue number | Description |
---|---|---|
2024-07-19 | SPL-259311, SPL-263863, SPL-263864, SPL-263865, SPL-263866 | Delayed creation of knowledge bundle |
2024-06-27 | SPL-258317, SPL-256878 | null byte injection in admin handler based endpoints |
2024-06-07 | SPL-257082, SPL-255939 | Crashing thread: TcpOutEloop and Shutdown on the Heavy Forwarder |
2024-06-07 | SPL-257081, SPL-255939 | Crashing thread: TcpOutEloop and Shutdown on the Heavy Forwarder |
2024-06-07 | SPL-257080, SPL-255939 | Crashing thread: TcpOutEloop and Shutdown on the Heavy Forwarder |
2024-05-24 | SPL-256405, SPL-265289, SPL-265290, SPL-265291, SPL-265292, SPL-265293, SPL-265294, SPL-265295, SPL-265296, SPL-265297 | HTTPServer does not read allowEmbedTokenAuth after certificate rotation / web server reload |
2024-05-21 | SPL-256104 | Maximum daily volume for a pool displayed as Unlimited, when license maximum typed in manually in 'A specific amount' field Workaround: When setting up maximum daily volume for this pool, choose 'The license maximum' option. |
2024-03-13 | SPL-252573, SPL-251434 | Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment |
2024-03-13 | SPL-252571, SPL-251434 | Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment |
2024-03-13 | SPL-252572, SPL-251434 | Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment |
2024-02-01 | SPL-250493, SPL-235583 | Metadata files inside buckets can be corrupted and left unsearchable. Workaround: n/a |
2023-12-10 | SPL-248188, SPL-248140 | Slow indexer detection calculate send queue bytes |
2023-12-10 | SPL-248187, SPL-248140 | Slow indexer detection calculate send queue bytes |
2023-11-07 | SPL-246765, SPL-245974 | HTTP Event Collector s2s endpoint ignores all inputs.conf.spec. |
2023-11-07 | SPL-246766, SPL-245974 | HTTP Event Collector s2s endpoint ignores all inputs.conf.spec. |
2023-11-03 | SPL-246640 | web.conf server.socket_host no longer overrides splunk-launch.conf SPLUNK_BINDIP Workaround: No workaround available. |
2023-09-25 | SPL-245071, SCP-64986 | Splunk Assist causes excessive logging before activation, sometimes on instances that do not run Splunk Assist at all Workaround: Disable Splunk Assist fully on those instances, see "Turn off Splunk Assist" in the Splunk Documentation for the procedure: https://docs.splunk.com/Documentation/Splunk/9.1.1/DMC/ActivateAssist#Turn_off_Splunk_Assist Modular inputs can be disabled individually in $SPLUNK_HOME/etc/apps/splunk_assist/local/inputs.conf |
2023-05-08 | SPL-239663 | Search History uses All Time range |
2023-05-07 | SPL-239645 | Cascading bundle replication stops due to an outage of an indexer and SH Captain does not generate new KO bundles till restarted or captaincy is transferred to another SH member Workaround: restart captain or transfer captaincy to another SHC member |
2022-11-14 | SPL-232803 | Job endpoint /services/search/jobs not returning QUEUED jobs Workaround: Queued job displays using job endpoint with SID: | rest /services/search/jobs/1668102339.174_23558BC9-6A39-4F4A-9FD2-968C358489B7 splunk_server=local |
2022-04-06 | SPL-222105 | When all inherited roles are taken out from admin role, it will cause admin user failed to show other users even though all capabilities is set natively. Workaround: Two possible approaches: 1. Remove the option grantableRoles = admin from authorize.conf - this is not permanent workaround and will need to be done every time admin role is modified. 2. Add any capabilities that the other user roles have to the "admin" role. |
2021-04-24 | SPL-204740, SPL-204735 | Deletion of a workload pool is allowed if there is a 'disabled' rule that is related to that workload pool and this can cause errors if the rule is re-enabled later Workaround: To prevent this issue: When you delete a workload pool, please make sure that you delete any disabled workload rules that are associated with that workload pool. To resolve the issue if you encounter this: Disable or delete the workload rule that is associated with a workload pool that does not exist anymore. |
2021-03-19 | SPL-202682 | The license usage report tab name is Previous 60 days, but the reports run over the last 30 days |
2020-08-10 | SPL-193389 | Parallel upload is not supported in gcp-sse-kms encryption mode Workaround: In the volumes using gcp-sse-kms encryption mode, specify "remote.gs.upload_chunk_size = 0" to disable parallel upload. |
2020-07-30 | SPL-192936 | Subsecond search - When you update metric.timestampResolution via the UI, it is not updated on the search head index.conf file. This does not affect search functionality. |
2019-10-03 | SPL-177447 | Bundle replication takes longer than expected time for indexers that have bundleEnforcerBlacklist configured |
2019-09-26 | SPL-177144, SPL-177326 | Under heavy search workload, the search memory usage estimation may be higher than actual usage |
2019-09-25 | SPL-177008, SPL-176710, SPL-177009 | Workload management fails to enable for addition of a pool with 1% cpu and 1% memory |
2019-09-16 | SPL-176514 | Offline rebuild of unsearchable bucket may lead to stale information in dbinspect searches |
2019-09-13 | SPL-176447 | SmartStore: Migration uploads of auto_high_volume buckets can fail indefinitely due to an XFS bug Workaround: Before migration, lower the max_concurrent_uploads setting in server.conf to 2. After migration, revert the setting to the default of 8. |
2019-07-19 | SPL-173449, SPL-173259 | timezone isn't stored for start_time/end_time of rule schedule every_day/every_week/every_month |
2019-03-26 | SPL-168314 | SmartStore standalone instance + Monitoring Console: Bootstrapping panel needs to reflect the standalone bootstrapping process |
2018-03-20 | SPL-152330, SPL-151992 | After installing Splunk on Windows using msiexec and the "GENRANDOMPASSWORD=1" option (and if generated password ends with backslash) admin is unable to login with msg "No users exist. Please set up a new user." Workaround: Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk [user_info] |
2017-06-29 | SPL-142789, SPL-95144 | Indexed message for Windows security event logs shows "FormatMessage error" Workaround: Splunk believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service. |
2017-05-09 | SPL-141693 | DataModel Editor - when child object has same name as inherited field, inherited field does not show in the inherited fields list. |
2017-01-06 | SPL-134707 | Splunk restart does not create missing server.pem certificate on Windows Workaround: Use bin/splunk createssl server-cert -d etc/auth/ -n server to generate a new certificate. |
2016-08-31 | SPL-127800 | Opting in to data sharing on a monitoring console produces duplicate data |
2016-07-26 | SPL-125052 | Sole Admin can demote themself to Power without path of recovery in GUI. Workaround: Through the command line, you can open notepad and modify the password file to regain 'Admin' status. |
2016-06-21 | SPL-123174 | JSON indexed_extractions doesn't work for TCP inputs |
Splunk Analytics for Hadoop
Date filed | Issue number | Description |
---|---|---|
2017-04-04 | ERP-2040 | Splunk archiving fails for large block sizes (buckets) due to HDFS write crashes for Hadoop version 2.8, 2.7.x Workaround: Upgrade Hadoop to 2.8.2 or higher. |
2015-09-09 | ERP-1650 | timestamp data type not properly deserialized. |
2015-08-05 | ERP-1619 | Searching on a newly created archive index before the bucket copy saved search is run causes a filenotfound exception. Workaround: Reenable the bucket copy saved search and let it run, or force the archiving to happen via | archivebuckets force=1 and then rerun the search. |
2015-07-07 | ERP-1598 | minsplit rampup - splits generation takes too long. Workaround: Set minsplits=maxsplits |
2015-05-12 | ERP-1502 | Non-accelerated pivot search on Pivot UI page waits for a long time to return result. |
2015-01-08 | ERP-1343, SPL-95174 | Splunk Analytics for Hadoop searches fail on corrupted journal.gz files, although Splunk searches run without error. Workaround: Add the journal.gz to the input path's blacklist (vix.input.1.ignore = ....) |
2014-10-27 | ERP-1216 | Data Explorer preview does not honor existing sourcetypes for big5/sjis files. |
2014-10-03 | ERP-1164 | Report acceleration summary gets deleted when two Splunk Analytics for Hadoop instances point to the same Splunk working directory. Workaround: To mitigate this issue, make sure that vix.splunk.home.hdfs (or Working directory in the UI) is unique on both search heads that are not in a pool. To keep your instances in the same working directory, configure vix.splunk.search.cache.path to be unique on both search heads. |
Welcome to Splunk Enterprise 9.2 | Increased skipped search rate after upgrade to 9.0 |
This documentation applies to the following versions of Splunk® Enterprise: 9.2.0
Feedback submitted, thanks!