Known issues
The following are issues and workarounds for this version of Splunk Enterprise.
Issues are listed in all relevant sections. Some issues appear more than once.
Refer to System requirements in the Installation Manual for a list of supported platforms and architectures.
For a list of deprecated features and platforms, refer to Deprecated features and removed features in this manual.
Upgrade issues
| Date filed | Issue number | Description |
|---|---|---|
| 2023-09-12 | SPL-244559, SPL-245880, SPL-246158 | Upgrades of Splunk Enterprise to version 9.1.x can fail on machines that run an operating system that does not use Unicode Transformation Format - 8 bit (UTF8) character set encoding. Workaround: This notice previously applied to Windows Server using the Japanese system locale. It now applies to any operating system that does not use UTF-8 for character set encoding. Prior to an upgrade, edit the $SPLUNK_HOME/etc/splunk-launch.conf file (%SPLUNK_HOME%\etc\splunk-launch.conf on Windows) and add the following line: PYTHONUTF8=1 Save the file, then perform the upgrade. If the upgrade succeeds, but Splunk Web then displays a blank screen when you attempt to log into the Splunk Enterprise instance, follow this workaround, then restart the Splunk Enterprise instance. |
| 2020-11-09 | SPL-197140, SPL-234386 | UF failed to start on Solaris 11.3 with error: "symbol in6addr_any: referenced symbol not found" Workaround: 1. Do not upgrade past Splunk 8.0.5 on Solaris 11.3 OR
2. Upgrade to Solaris 11.4 |
| 2020-08-31 | SPL-194426 | External search command chunked v2 python SDK fails with multibyte result data under python 3. Workaround: Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters. App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps. Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available. |
| 2020-07-10 | SPL-191850 | The .deb installation package will fail if dpkg version doesn't support an .xz compressed control file. Workaround: Update dpkg to version 1.17.6 or later. |
| 2018-04-13 | SPL-153403 | After running the "clean userdata" command, admin is unable to login with msg "No users exist. Please set up a new user." Workaround: Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk [user_info] |
Data input issues
| Date filed | Issue number | Description |
|---|---|---|
| 2024-01-13 | SPL-249543, SPL-251748, SPL-251749, SPL-253929, SPL-251746, SPL-253927, SPL-253928 | TcpInputProcessor not able to drain splunktcpin queue during graceful shutdown. Workaround: Splunk recommends customers set `useACK` to true to ensure in-memory is not dropped in the event of indexer rolling restarts or repaving. Thus, the best short-term solution is to set `useACK` to `true`. |
| 2024-01-10 | SPL-249424, SPL-249409 | Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x |
| 2024-01-10 | SPL-249423, SPL-249409 | Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x |
| 2024-01-10 | SPL-249422, SPL-249409 | Splunk UF (windows) huge amount of duplicating logs due to re-reading log file after Upgrade to 9.1.0.1 Workaround: No workaround. Downgrade to 8.2.x |
| 2023-11-07 | SPL-246769, SPL-243845 | HTTP Input HEC input ignores _meta in inputs.conf |
| 2023-11-07 | SPL-246770, SPL-243845 | HTTP Input HEC input ignores _meta in inputs.conf |
| 2023-11-07 | SPL-246768, SPL-243845 | HTTP Input HEC input ignores _meta in inputs.conf |
| 2023-06-15 | SPL-241076, SPL-251249, SPL-251251, SPL-251329, SPL-251250 | Metrics event can be indexed in default event index when mcollect is used. Workaround: Avoid to restart if queue is blocked. (wish maybe should be best practise but the revert is what always happen customer queue blocked let restart to solve it .) |
| 2022-09-08 | SPL-229853, SPL-229208 | PowerShell Modular input stopped working after UF 9.0 upgrade |
| 2022-08-09 | SPL-228117, SPL-257140 | "file" is incorrectly listed as a supported scheme for ingest actions in outputs.conf.spec |
| 2022-07-12 | SPL-226624, SPL-226408 | SmartStore configuration of AWS S3 bucket is rejected with IAM credentials and a custom endpoint |
| 2022-06-30 | SPL-226381, SPL-221387 | When a non-IAM user sends only `path` as param the UI become non-responsive for approximately 4 mins |
| 2022-04-08 | SPL-222366 | Ingest Actions does not work with Splunk Free, Personalized Devtest, Developer, and Forwarder-only licenses |
Search issues
| Date filed | Issue number | Description |
|---|---|---|
| 2024-05-14 | SPL-255737 | Version 2 of the stats command can't distinguish prestats and non-prestats data in summary index at the event level. |
| 2023-09-27 | SPL-245135, SPL-245127 | Indexer Search crash with no back-trace in PCRE2 on X86_64 Workaround: Re-running the search is the only workaround. This issue is not deterministic within a search, so re-running will usually work. |
| 2023-09-25 | SPL-245076, SPL-225303, SPL-245054 | Segfault in BucketSummaryActorThread, originating in the lookup processor, causes crashing search processes for Data Model Accelerations Workaround: Update the `max_memtable_bytes` in my the limits.conf/[lookup] stanza to a large value, such as 2147483648 (2GB), to increase the threshold at which lookup tables are indexed to disk on the indexers instead of using in-memory indexes for those lookups. We recommend sizing the threshold based on lookups in your environment. Please note that this will increase the amount of memory used for searches that use lookups, which could be significant in environments with many concurrent searches. |
| 2023-06-28 | SPL-241609, SPL-227018 | In rare cases in some buckets, searches can return some empty field values or missing events for indexed fields when the bucket contains small metadata files (between 4-8KB) leading to "not all cwpairs were found" in search.log Workaround: Within search.log in the dispatch directory of a failed search, look for the term "not all cwpairs found". If this is found, upgrade to a version that fixes the issue. |
| 2023-04-26 | SPL-239409 | The 'sendemail' command no longer honors the field order from search results |
| 2023-04-14 | SPL-238738 | Federated Search for Splunk does not support the "Show Source" Field Action |
| 2023-03-28 | SPL-237902 | Ad hoc searches that specify earliest relative time offset assuming from 'now' should explicitly include 'latest=now' to avoid a potential time range inaccuracy Workaround: Ad hoc searches searches that use the earliest time modifier with a relative time offset should also include 'latest=now' in order to avoid time range inaccuracies. For example, if you want to get all events from the last 10 seconds starting at 01:00:10, the following search returns all events that occur between the time of 01:00:00 and 01:00:10, as expected. index=main earliest=-10s latest=now Running the same search without including 'latest=now' might produce unpredictable results or impact performance in certain scenarios when the search head is overloaded with ad hoc searches. See Specify earliest relative time offset and latest time in ad hoc searches in the Splunk platform Search Manual. |
| 2022-10-31 | SPL-232219 | Resolve event count issues for Tstats in Standard mode when append=true |
| 2022-10-24 | SPL-231946, SPL-233842, SPL-233843 | |metadata command ignores splunk_server parameter Workaround: Use other search commands like |tstats instead of |metadata if you need to filter on splunk_server |
| 2022-10-20 | SPL-231830, SPL-239319, SPL-239320 | SearchJob sometimes fails and returns error "Search <ID> not found. The search may have been cancelled while there are still subscribers" Workaround: Remark : this Splunk Enterprise issue may impact ITSI UI with loading issues (KPI and thresholds preview, Share Base search validation, Entity import, Maintenance windows preview) Fix : upgrade Splunk to a fixed version |
| 2022-09-26 | SPL-230682 | Tstats returns incorrect event counts when using append=true |
| 2022-09-23 | SPL-230581, SPL-227411 | After upgrade one set of clustered indexers has increase in thread crashes during search |
| 2022-09-13 | SPL-230091, SPL-231852 | Search can use large amount of memory on large/malformed events that (look like) XML |
| 2022-09-12 | SPL-229969, SPL-230549, SPL-230857, SPL-230859 | regex doesn't honor the caret symbol ^ (start of string) in some conditions. |
| 2022-09-08 | SPL-229882 | Job inspector can no longer show search logs when the logs are titled search.log.1, search.log.2, etc. Workaround: Retrieve the log file from the filesystem. The search's dispatch directory is located at $SPLUNK_HOME/var/run/splunk/dispatch/<sid>. Dispatch directories can also be collected in a Splunk Diag if the search's time-to-live (TTL) has not yet expired. |
| 2022-08-29 | SPL-229278 | Search crashes with "StatsBuffer found inconsistent row" after upgrading Workaround: Use '| noop feature_flag=stats:allow_stats_v2:false' in a search to use StatsV1 to avoid this issue for a single search. Alternatively, add '[stats] use_stats_v2 = false' in the limits.conf file to globally configure this setting for all searches. Both workarounds impact the performance for searches using the 'stats' family of functions such as 'chart', 'timechart', 'eventstats', 'tstats', 'prestats', 'mstats' and 'streamstats' because performance enhancements added with StatsV2 are not used. As a result, upgrading to Splunk Enterprise version 9.0.1 or higher is preferred. |
| 2022-08-02 | SPL-227728, SPL-226351 | Mcatalog subsearch hitting maxout limit preventing metric rollup from populating results correctly. |
| 2022-07-29 | SPL-227633 | Error : Script execution failed for external search command 'runshellscript' Workaround: The setting precalculate_required_fields_for_alerts=0 can be set on saved searches that have no other alert actions attached aside from the "Run A Script" action, to quash the error. For saved searches that have multiple alert action attached, this may not be safe as it will disable back propagation of required fields for all alert actions, which might result in the parent search extracting more fields than required, which could negatively impact performance for that search. |
| 2022-07-28 | SPL-227547 | Fix hash algorithm for numbers and handle hash collisions for lookups Workaround: Add allow_caching=f to the lookup command: | lookup <name> allow_caching=f ... On 7.3+: Add allow_caching=f to the lookup definition on the search head transforms.conf: [<lookup name>] allow_caching = f To check if you might be running into this issue, you'll need to enable debug on the search in question by adding: | noop log_DEBUG=CachedProvider <pre> If you have hits for the cached lookup, like in the sample log below, you can hit this issue. <pre> DEBUG CachedProvider - Cached provider metrics: lookup=<lookup name> hits=67064 misses=321 total=67385 |
| 2022-06-23 | SPL-226021, SPL-224456 | stash_new files for apps using summary indexes, like ITSI, are not being deleted from splunk/var/spool/splunk because TailReader is hung Workaround: restart the search head clears the stash_new files |
| 2022-06-23 | SPL-226017, SPL-176333 | Lookups may return incorrect results due to internal caching Workaround: Add allow_caching=f to the lookup command: | lookup <name> allow_caching=f ... On 7.3+: Add allow_caching=f to the lookup definition on the search head transforms.conf: [<lookup name>] allow_caching = f To check if you might be running into this issue, you'll need to enable debug on the search in question by adding: | noop log_DEBUG=CachedProvider <pre> If you have hits for the cached lookup, like in the sample log below, you can hit this issue. <pre> DEBUG CachedProvider - Cached provider metrics: lookup=<lookup name> hits=67064 misses=321 total=67385 |
| 2022-06-21 | SPL-225955, SPL-223099 | Job completion emails aren't received while sending the job to background |
| 2022-05-25 | SPL-224816, SPL-232036, SPL-232477 | Standard mode federated searches of accelerated data models with 'tstats' fail or produce unexpected behavior when 'prestats=t' Workaround: possibly: - run the search in the right order |
| 2021-12-21 | SPL-216787 | Searches are cancelled or time out when the user leaves the browser window or switches tabs. Workaround: In Splunk Enterprise 8.1.7, 8.2.4, and higher change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62.Details
|
| 2021-09-22 | SPL-212495, SPL-196040, SPL-219811 | Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles Workaround: none |
| 2021-03-08 | SPL-202077, SPL-176333 | Lookups may return incorrect results due to internal caching Workaround: Add allow_caching=f to the lookup command: | lookup <name> allow_caching=f ... On 7.3+: Add allow_caching=f to the lookup definition on the search head transforms.conf: [<lookup name>] allow_caching = f To check if you might be running into this issue, you'll need to enable debug on the search in question by adding: | noop log_DEBUG=CachedProvider <pre> If you have hits for the cached lookup, like in the sample log below, you can hit this issue. <pre> DEBUG CachedProvider - Cached provider metrics: lookup=<lookup name> hits=67064 misses=321 total=67385 |
| 2021-02-25 | SPL-201628 | `srchTimeWin` and `srchTimeEarliest` settings cannot be unset for the admin role. Workaround: Ensure that the admin role is not configured as "Unset" and is explicitly configured to either no restriction or a restriction in the UI (Navigate to Edit Role > Resources > Role search time window limit), or through conf file authorize.conf under attribute name srchTimeEarliest. |
| 2020-12-06 | SPL-198314, SPL-233681, SPL-233762 | Exporting _time field applies user timezone offset but contains the server's timezone (usually +0000) Workaround: Force a specific time format by using strftime in an eval command. for example, add | convert timeformat="%FT%T.%3Q%z" ctime(_time) to the end of your search |
| 2020-12-04 | SPL-198284, SPL-231587 | Crash in search process in PrecacheUsersThread when max_searches_per_process is set lower than default Workaround: Set limits.conf back to default, by removing any override of max_searches_per_process. For example: [search] max_searches_per_process=1 to [search] |
| 2020-08-31 | SPL-194426 | External search command chunked v2 python SDK fails with multibyte result data under python 3. Workaround: Apps may experience this issue if they: implement a custom search command using the Splunk Enterprise SDK for Python between versions 1.6.5 and 1.6.13; are executed by Splunk Enterprise or Splunk Cloud using Python 3; and are sent events with multi-byte characters. App developers whose apps implement a custom search command using a version of the Splunk Enterprise SDK for Python must update to version 1.6.14 or higher and release new versions of their apps. Splunk Enterprise and Splunk Cloud administrators who are using apps impacted by this issue must update to app versions that use the Splunk Enterprise SDK for Python version 1.6.14 or higher. If this is not possible, administrators are encouraged to either: allow these apps to be executed using Python 2; or cease usage of impacted apps until updated versions are available. |
| 2020-02-12 | SPL-183259 | When generating LISPY for field values that are numbers (""), the values aren't deduplicated, which can cause slowdowns in certain scenarios Workaround: Dedup values in search before, for example: instead of index="field_test" [search index="field_test" globalCallID_callId=1234* | fields globalCallID_callId] add a stats or dedup in the subsearch: index="field_test" [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ] If that list is still large and you're seeing the slowdown, consider moving the filtering to a | where after the initial search, for example: index="field_test" globalCallID_callId=* | where [search index="field_test" globalCallID_callId=123* | stats values(globalCallID_callId) AS globalCallID_callId | mvexpand globalCallID_callId ] |
| 2020-01-10 | SPL-181573 | geostats provides incorrect results for lower zoom levels when split BY has a higher cardinality than globallimit. Workaround: - Increase globallimit to the value of "unique values" number mentioned in the warning message: "The split by field <field> has a large number of unique values <number>. Chart column set will be trimmed to 10. Use globallimit argument to control column count." - Use very high globallimit in geostats and post process after if needed - Don't use BY in geostats - Use lower cardinality BY and/or higher globallimit in geostats |
| 2017-07-13 | SPL-143111 | "Splunkd daemon is not responding" when edit local windows event log collection |
| 2017-04-04 | SPL-140765 | Splunk having problems extracting json file consisting of 68k plus key-value pairs |
| 2016-11-29 | SPL-133182 | When two datasets have identical names but one is local (private) while the other is global, attempts to view or extend the global dataset use results from the local dataset instead. |
| 2014-10-02 | SPL-91638, SPL-107375 | For scheduled searches in a search head cluster, empty search jobs may appear in the job inspector for a cluster member. |
Federated search issues
| Date filed | Issue number | Description |
|---|---|---|
| 2024-05-23 | SPL-256393, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
| 2024-05-23 | SPL-256394, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
| 2024-05-23 | SPL-256392, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
| 2024-05-23 | SPL-256391, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
| 2024-05-23 | SPL-256390, SPL-251563 | Meraki onPrem fails to send some bundles to meraki-security cloud. The proxy bundles are actually processed incorrectly on the the cloud members instead of outright being rejected |
| 2024-04-23 | SPL-254718, SPL-253248, SPL-255069 | Federated searches not completing with error "Socket error during transaction. Socket error: Success" |
| 2024-04-05 | SPL-253755, SPL-252488, SPL-253757 | federated search should alert ( and block the search ) when it is run in realtime mode |
| 2024-03-27 | SPL-253248, SPL-254718, SPL-254719, SPL-254720, SPL-254722, SPL-254721 | Federated searches not completing with error "Socket error during transaction. Socket error: Success" |
| 2024-03-26 | SPL-253175, SPL-244551 | Federated search failures seen on RSH due to terminated search connections from FSH |
| 2024-03-12 | SPL-252488, SPL-248786, SPL-253755 | Lookups in transparent mode don't use proper lookup when fsh and rsh have lookup with same name |
| 2024-02-23 | SPL-251536 | Block edit request to change existing federated provider mode and useFSHKO settings |
| 2024-01-18 | SPL-249666, SPL-244551 | FS-StandardMode : Standalone sub-search with HEAD doesn't return any results |
| 2023-12-21 | SPL-248786, SPL-252486, SPL-252487, SPL-252488 | Lookups in transparent mode don't use proper lookup when fsh and rsh have lookup with same name Workaround: If the `rsh` is getting transparent searches and it does not have indexers connected to it, the `rsh` does not look on the kvstore values that were sent to it from the `fsh`. The workaround is to have indexers attached to the `rsh` |
| 2023-09-05 | SPL-244248, SPL-239298 | Federated Search, Enterprise --> Cloud configuration: Performance degradation increases when the number of indexers increases in the RSH Workaround: One possible workaround is to use a more efficient query. For example, use "| tstats count where index=main by splunk_server" instead of "index=main | stats count by splunk_server".
Use this workaround if you are using your Splunk Enterprise federated search head (FSH) instance only for running federated searches. This workaround might affect non-federated searches. On the Splunk Enterprise FSH, follow these steps:
|
| 2023-07-20 | SPL-242282, SPL-242864 | Federated Searches fail for union commands when query optimization diverge between FSH x RSH |
| 2023-07-12 | SPL-242049, SPL-248189, SPL-248311, SPL-248312 | Kvstore files are not converted to csv files in the bundles when local indexers are not present even when remote providers are present Workaround: If an indexer ( a distributed peer ) is added to the local deployment ( the federated search head ), then the issue is resolved. On deployment that don't have indexers at the moment a low performance indexer should be created ( on a vm etc ) and added as a distributed peer. |
| 2023-06-26 | SPL-241446 | rsh_sid should be logged in the fsh audit log as well |
| 2023-05-22 | SPL-240242, SPL-262206 | Federated Search: When exporting results, the remote search head (RSH) returns exceptions when it sees federated search head (FSH) socket errors. The RSH should ignore FSH socket errors. |
| 2023-05-02 | SPL-239436 | In standard mode federated search, outputlookup existence check on RSH causes search to terminate early although it is not run on RSH Workaround: Define the lookup on both federated search head and remote search head. |
| 2023-04-17 | SPL-238767, SPL-244936, SPL-244937 | Standard mode federated search with longer than a minute From command searches, encounters socket ReadWrite error when the federated provider points to a cloud Load balancer, due to idle timeout on the LoadBalancer config Workaround: If you encounter this issue, update the federated provider definition (created on the federated search head in Splunk Web), so that its Remote Host points to a remote deployment cluster member instead of to the remote deployment cluster load balancer. |
| 2023-04-14 | SPL-238738 | Federated Search for Splunk does not support the "Show Source" Field Action |
| 2023-04-11 | SPL-238512, SPL-239266 | Federated search UI does not support mapping federated indexes to data model datasets that have dot characters in their names |
| 2023-04-10 | SPL-238501 | Federated search "outputlookup" command cannot add data to local lookup table Workaround: Define the same lookup on the remote search head too, so the remote search head will not error out early and return 0 results. |
| 2023-03-30 | SPL-238029, SPL-239359, SPL-239360 | Standard mode federated search - A multistats search with a tstats subsearch where prestats=t and a federated index is used as a data model throws an error. |
| 2023-03-28 | SPL-237883, SPL-239361, SPL-239362 | Transparent Mode federated search - Using table and stats in the same federated search causes the search to return empty results , when run in smart or fast mode Workaround: Run searches in VerboseMode to ensure all fields are returned to the FSH. |
| 2023-03-24 | SPL-237796, SPL-248319 | In transparent mode Federated Search for Splunk, the makeresults command returns more rows than expected Workaround: Convert all occurrences of makeresults to makeresults | head 1.If you need more results, change the head command parameter accordingly. For example,makeresults count=5 would become makeresults count=5 | head 5. |
| 2022-12-06 | SPL-233685, SPL-214007, PSRT-3936 | Federated search: An HTTP parser error causes searches of saved search datasets with long queries to end abruptly |
| 2022-10-31 | SPL-232219 | Resolve event count issues for Tstats in Standard mode when append=true |
| 2022-10-19 | SPL-231712 | Create/Edit Role - In the UI, the "Wildcards" tool cannot be used to specify allowed federated indexes for standard mode federated search |
| 2022-09-26 | SPL-230682 | Tstats returns incorrect event counts when using append=true |
| 2022-08-23 | SPL-228969 | Federated Search: In Splunk Web federated index UI you cannot provide data model Dataset Name values that contain a dot ( . ) character Workaround: This is a limitation for users of standard mode federated search who want to set up federated indexes that map to data model datasets. It means that such users cannot set up federated indexes for data model datasets that are subordinate to a root dataset. For example, if the root data model dataset is Network_Traffic, you cannot map a federated index to the subordinate data model dataset Network_Traffic.All_Traffic.As a workaround, users can run tstats searches that use the nodename argument to filter out data that does not belong to a specific data model dataset: | tstats ... where nodename=Network_Traffic.All_Traffic. |
| 2022-07-27 | SPL-227530, SPL-234988 | Splunk-to-Splunk federated search: After upgrade, the remote search head gets stuck in a loop of transferring proxy bundles to the remote indexers and failing Workaround: To stop a proxy bundle (pb_t1) from being sent endlessly from the remote deployment to it's indexers, you need to ensure that the local deployment generates a new bundle. Once this one new bundle is generated and sent to the remote deployment (as (pb_t2), the remote deployment will stop sending the previous bundle to the indexers. To make sure that happens:
1. There must be a valid transparent mode federated provider definition that connects the local deployment to the remote deployment that keeps sending the proxy bundle.
2. You can create a dummy tag on the local deployment to trigger the bundle replication from the local deployment to the remote deployment.
3. Trigger a search from the local deployment. This will make sure that the bundle is sent ASAP towards the remote deployment. |
| 2022-07-15 | SPL-226877 | Federated Search UI Error: Cannot create saved search dataset for federated index if dataset name contains space Workaround: Use REST API to create the federated saved search instead:
|
| 2022-06-23 | SPL-226038 | In a transparent mode federated tstats search of an accelerated data model that is located only on the FSH, results are returned only from the FSH, not the RSH, when summariesonly=t |
| 2022-06-21 | SPL-225949 | federated.conf.spec and federated.conf.example files are missing from the product build Workaround: Modify 'federated.conf.spec' and 'federated.conf.example' with the contents on the below document: https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Federatedconf To check the workaround is working, restart the instance to see if logs appear. (See attached image: Image-20221110-234300.png) |
| 2022-06-16 | SPL-225826 | Transparent mode federated searches over accelerated data model datasets cannot return remote results because their summaries are not present on the RSH |
| 2022-05-31 | SPL-225037 | Remote dataset dropdown menu resets to "Index" after selecting federated provider |
| 2022-05-25 | SPL-224816, SPL-232036, SPL-232477 | Standard mode federated searches of accelerated data models with 'tstats' fail or produce unexpected behavior when 'prestats=t' Workaround: possibly: - run the search in the right order |
| 2022-03-09 | SPL-220289, SPL-245017 | Federated Search Transparent Mode: Commands that have subsearches like join and append may result in failures on RSH due to missing application context Workaround: If the search is being run in an application context that does not exist on the remote deployment, install the missing application on the remote deployment. |
| 2022-02-25 | SPL-219793 | Some commands in federated searches return incorrect resultCount values when run in verbose mode Workaround: Use Verbose and Smart mode specifically for searches with transforming commands like stats, chart, and timechart, and then review the results in the Statistics tab. To review event counts, run non-transforming searches in Fast mode.Search-time field extraction usually requires searches without transforming commands that run in either Verbose or Smart mode. When you run searches in Fast mode, you can ensure that search-time field extraction takes place for federated searches by appending |
| 2022-02-08 | SPL-218842, SPL-252272, SPL-242740 | Some reporting commands in federated search return incorrect eventCount Workaround: Use Verbose and Smart mode specifically for searches with transforming commands like stats, chart, and timechart, and then review the results in the Statistics tab. To review event counts, run non-transforming searches in Fast mode.Search-time field extraction usually requires searches without transforming commands that run in either Verbose or Smart mode. When you run searches in Fast mode, you can ensure that search-time field extraction takes place for federated searches by appending |
| 2022-02-08 | SPL-218841 | Reporting command in verbose mode returns 0 events despite correct event_count |
| 2021-10-14 | SPL-213745, SPL-251131 | Standard mode federated search: Unable to set federated index as default index |
Saved search, alerting, scheduling, and job management issues
| Date filed | Issue number | Description |
|---|---|---|
| 2023-11-08 | SPL-246785, SPL-244383 | Search-Scheduler Splunk Crashes on Job Servers in SHC. Workaround: Workaround collections.conf [LoggedOutSessionTokens] disabled = true server.conf
[general]
invalidateSessionTokensOnLogout = false |
| 2022-06-29 | SPL-226269 | $SPLUNK_HOME/etc/apps/search/local/alert_actions.conf is created when an email is sent via an alert Regression (SPL-194487) Workaround: manually remove the show_password = true from \[email] under $SPLUNK_HOME/etc/apps/search/local/alert_actions.conf |
| 2022-06-21 | SPL-225955, SPL-223099 | Job completion emails aren't received while sending the job to background |
| 2022-03-09 | SPL-220208, SPL-228900, SPL-223021, SPL-223025, SPL-226444, SPL-228628 | Summary Director consuming excessive amounts of CPU and RAM in certain situations |
| 2019-09-20 | SPL-176812 | Multiple SH Clustering with single deployer can't use datamodel summary sharing |
| 2018-09-19 | SPL-160286 | The data preview for the Add Data workflow does not display for Log to Metrics source types |
| 2017-11-29 | SPL-146802 | Distributed environment requires index defined on search head for log event alerts |
| 2017-08-14 | SPL-143947 | Report acceleration is broken for users with a configured role-based access filter |
Charting, reporting, and visualization issues
| Date filed | Issue number | Description |
|---|---|---|
| 2023-06-08 | SPL-240750 | Inconsistency in displayed timezone in Dashboard Studio when using time range tokens |
| 2023-02-21 | SPL-236371, SPL-228658 | "Down Arrow" of chart legend scrolling does not work Workaround: To use another format, but rejected |
| 2023-01-26 | SPL-235420 | Link to Dashboard show first 30 apps |
| 2022-10-20 | SPL-231838 | Form fieldset input choice strings are not localized |
| 2022-08-04 | SPL-227909, SPL-228844 | Inconsistent behavior for new dashboard with showing/not showing data/search results from default token |
| 2022-07-21 | SPL-227157, SPL-226132 | User with format phxxxx cannot open a dashboard on 9.x |
| 2022-06-29 | SPL-226337, SPL-224661 | Custom table cell renderer doesn't work consistently on refresh or when switching back from Edit -> Source view Workaround: refresh page |
| 2022-04-26 | SPL-223193, SPL-233133 | "Open in Search" function doesn't work with chained searches in Dashboard Studio when the time range depends on an input/token, showing error "Invalid earliest_time" |
| 2022-03-29 | SPL-221489, SPL-222825, SPL-222826 | Find search bar in Splunk toolbar only returns Classic dashboards |
| 2016-04-27 | SPL-118911 | In SHC, referenced saved real-time searches in a dashboard do not stream results. Workaround: See Troubleshoot referenced real-time searches for workaround details.
|
Distributed search and search head clustering issues
| Date filed | Issue number | Description |
|---|---|---|
| 2022-07-26 | SPL-227395, SPL-228155 | Deployer push is taking longer time Workaround: +No workaround found so far on 9.0.0 |
| 2022-07-18 | SPL-227012, SPL-225689 | crashed in NewTransamProcessor NewTransam.cpp:290: bool NewTransaction::isCompatible(SearchResultWrapper&, bool&): Assertion `_opened' failed. |
| 2022-04-20 | SPL-222917, SPL-230428 | Crash in indexer discovery service on search head |
| 2022-03-22 | SPL-221130, SPL-224931, SPL-225711 | Search head clustering - intermittent "Splunk Cloud" logo shown on splunkweb and "UNKNOWN_VERSION" Splunk version returned Workaround: Customers can verify whether their environment is affected with following SPL against their SHs: index=_internal host IN (<CommaSeparatedSHList>) source=*web_service.log* "Splunk appserver version=UNKNOWN_VERSION build=000" Refreshing the browser tab will temporarily resolve the issue. No root cause/fix has been identified yet. |
| 2021-09-22 | SPL-212495, SPL-196040, SPL-219811 | Excessive logging 'WARN SearchResultsFiles Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"' for SearchResultsFiles Workaround: none |
| 2021-03-26 | SPL-203060 | The splunkd process changes the local distsearch.conf on service start Workaround: There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:
|
| 2017-11-29 | SPL-146802 | Distributed environment requires index defined on search head for log event alerts |
| 2017-03-13 | SPL-138654 | Splunk searches fail when filepath gets too long on Windows |
| 2016-07-12 | SPL-124085 | On Search Head Cluster It is not possible to remove an App from the SHs once it has been disabled. |
Data model and pivot issues
| Date filed | Issue number | Description |
|---|---|---|
| 2022-12-15 | SPL-234152, SPL-223333 | Error in 'mvexpand' command: Invalid argument: '...' when trying to do mvexpand on a fieldname with a space |
| 2022-03-09 | SPL-220208, SPL-228900, SPL-223021, SPL-223025, SPL-226444, SPL-228628 | Summary Director consuming excessive amounts of CPU and RAM in certain situations |
| 2019-09-20 | SPL-176812 | Multiple SH Clustering with single deployer can't use datamodel summary sharing |
Indexer and indexer clustering issues
| Date filed | Issue number | Description |
|---|---|---|
| 2024-01-13 | SPL-249543, SPL-251748, SPL-251749, SPL-253929, SPL-251746, SPL-253927, SPL-253928 | TcpInputProcessor not able to drain splunktcpin queue during graceful shutdown. Workaround: Splunk recommends customers set `useACK` to true to ensure in-memory is not dropped in the event of indexer rolling restarts or repaving. Thus, the best short-term solution is to set `useACK` to `true`. |
| 2022-08-18 | SPL-228672, SPL-231396 | validation of bundle returns "restart required" always on any app when there is a password field with encrypted bundles |
| 2022-07-02 | SPL-226423, SPL-226596, SPL-226662, SPL-226829 | Indexer cluster bundle status stuck in "Bundle Creation is in progress" following error to apply the bundle with message "User '<name>' with roles { <roles> } cannot write". Workaround: Make all encrypted fields writable in the metadata/local.meta files inside etc/manager-apps/ (so, where you find "write : [ somerole ]", you might change it to "write : [ * ]" or "write : [ role_that_user_applying_bundle_has ]" |
| 2022-03-28 | SPL-221431, SPL-216614 | Searchable Rolling Restart stuck reassigning primacy when indexers take more than streaming_replication_wait_secs to roll their buckets when being decommissioned. Workaround: Increase server.conf[clustering]streaming_replication_wait_secs to ensure all streaming targets will be completed. streaming_replication_wait_secs=1800 (a maximum wait of 30min) is not unreasonable, but the setting will affect all types of CM-initiated restarts, so make sure not to set it too high. |
| 2016-08-25 | SPL-127353 | Data rebalance finishes early when one peer is the source for all buckets Workaround: when only one indexer in a cluster indexed data (has all the searchable copies), rebalance once before adding the new indexer, and then rebalance a second time |
Universal forwarder issues
| Date filed | Issue number | Description |
|---|---|---|
| 2024-02-22 | SPL-251517, SPL-237849 | CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 Upgrade to version 9.0.0.1 issue doesn't appear. [2]
The detailed workaround didn't solve the issue. |
| 2024-02-22 | SPL-251515, SPL-237849 | CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 Upgrade to version 9.0.0.1 issue doesn't appear. [4]
The detailed workaround didn't solve the issue. |
| 2024-02-22 | SPL-251516, SPL-237849 | CHECK_METHOD = modtime not working as expected in ver. 9.0.4 upgrading from 8.2.7. Workaround: time_before_close=0 Upgrade to version 9.0.0.1 issue doesn't appear. [6]
The detailed workaround didn't solve the issue. |
| 2022-12-01 | SPL-233535, SPL-231086 | UF 9.x Unnecessary user creation during silent installation Workaround: Delete etc/passwd manually after installation, create user-seed.conf with new user in SPLUNK_HOME/etc/system/local and restart Splunk Customer claims that they were able to bypass the issue by using SPLUNKPASSWORD="" option during installation but I was not able to reproduce this
example of user-seed.conf:
https://drive.google.com/file/d/1GrypUL6719V0tGbw0Mv-u76jUYZ8LBS_/view?usp=sharing |
| 2022-10-20 | SPL-231793 | Crashing in TcpOutEloop thread with assertion_failure="_refCount > 0" Workaround: autoBatch=false |
| 2022-10-14 | SPL-231514, SPL-228406 | UF crash on EventLoop::run assert rv > 0 Workaround: N/A |
| 2022-09-08 | SPL-229853, SPL-229208 | PowerShell Modular input stopped working after UF 9.0 upgrade |
| 2022-07-30 | SPL-227653, SPL-231927 | UF throws erroneous WARN for KVSTORE SSL misconfiguration on startup - server.conf//sslVerifyServerCert or "Starting migrate-kvstore." Workaround: It's safe to ignore the warning or you can disable the kvstore explicitly with server.conf: [kvstore] disabled = true |
| 2022-07-13 | SPL-226795, SPL-222481, SPL-231443 | Splunk UF Windows Event Log Stopped Being Ingested |
| 2022-06-23 | SPL-226019 | Warning appears in the universal forwarder whenever any spl command is run: Warning: Attempting to revert the SPLUNK_HOME ownership Warning: Executing "chown -R splunk /opt/splunkforwarder". This warning is expected and will not affect functionality. |
| 2022-06-22 | SPL-226003, SPL-237740 | When forwarding from an 9.0 instance with useAck enabled, ingestion stops after some time with errors: "Invalid ACK received from indexer=" Workaround: As a workaround, disable useAck in outputs.conf on the forwarder. After disabling, indexers start to ingest data. If you need useACK to prevent data loss, disabling autoBatch in outputs.conf can remediate the issue too, but it impacts throughput - no worse than 8.x, but no improvement for 9.0. |
| 2022-06-06 | SPL-225379 | Ownership of files mentioned in manifest file is splunk:splunk instead of root:root after enabling boot start as root user for initd Workaround: When changing UF user, manually chown SPLUNK_HOME to the new user, including first time install/upgrade, or manually enable boot-start. |
| 2022-05-16 | SPL-224264, SPL-224265 | Splunk UF not starting on Debian 11 (x86_64 and arm64) |
| 2022-04-20 | SPL-222917, SPL-230428 | Crash in indexer discovery service on search head |
| 2020-11-09 | SPL-197140, SPL-234386 | UF failed to start on Solaris 11.3 with error: "symbol in6addr_any: referenced symbol not found" Workaround: 1. Do not upgrade past Splunk 8.0.5 on Solaris 11.3 OR
2. Upgrade to Solaris 11.4 |
Monitoring Console issues
| Date filed | Issue number | Description |
|---|---|---|
| 2023-09-14 | SPL-244687, SPL-249851 | Bucket Health Status is not cleared even after 24 hours until a new bucket is created |
| 2022-12-20 | SPL-234321, SPL-231388 | Backport for splunkd crashed in HealthDistIngestionLatency::calculateAndUpdateHealthColor() Workaround: In health.conf, use the following settings:
[feature:ingestion_latency] indicator:ingestion_latency_lag_sec:yellow = 0 indicator:ingestion_latency_lag_sec:red = 0 |
| 2022-11-02 | SPL-232361, SPL-227612, SPL-236561 | Custom Group selection box in Monitoring Console breaks/disappears on clicking for selection |
| 2021-03-29 | SPL-203100 | Summary page on monitoring console doesn't show correct RF/SF when not running on the CM. |
| 2019-11-13 | SPL-179528 | The splunktcp and splunktcp-ssl stanzas are not reloadable in inputs.conf |
| 2017-08-14 | SPL-143981 | Uninstall app dialog does not show the app name correctly when the app doesn't have the label |
| 2017-05-24 | SPL-141982 | Upload modal should use size=large File element |
| 2017-04-19 | SPL-141274 | Clicking Install multiple times in Install dialog causes error |
| 2016-11-14 | SPL-132151 | XML error when trying to download uninstalled app |
Splunk Web and interface issues
| Date filed | Issue number | Description |
|---|---|---|
| 2024-07-29 | SPL-259841 | Unable to install or update Splunkbase apps using the In-Product App Browser Workaround: On September 18th, 2024 Splunkbase certificates expire on Splunk Enterprise version 9.0.0, version 8.2.8 and lower, and version 8.1.10 and lower. To continue installing and updating Splunkbase apps using the In-Product App Browser (Find More Apps / Manage Apps), upgrade to a supported version of Splunk Enterprise. For instructions on how to upgrade, see https://docs.splunk.com/Documentation/Splunk/latest/Installation/HowtoupgradeSplunk |
| 2023-04-10 | SPL-238486, SPL-235850 | ui-prefs optimizations - Only use browser-based storage for ui-prefs |
| 2022-10-20 | SPL-231830, SPL-239319, SPL-239320 | SearchJob sometimes fails and returns error "Search <ID> not found. The search may have been cancelled while there are still subscribers" Workaround: Remark : this Splunk Enterprise issue may impact ITSI UI with loading issues (KPI and thresholds preview, Share Base search validation, Entity import, Maintenance windows preview) Fix : upgrade Splunk to a fixed version |
| 2022-10-20 | SPL-231838 | Form fieldset input choice strings are not localized |
| 2022-05-31 | SPL-225037 | Remote dataset dropdown menu resets to "Index" after selecting federated provider |
| 2021-12-21 | SPL-216787 | Searches are cancelled or time out when the user leaves the browser window or switches tabs. Workaround: In Splunk Enterprise 8.1.7, 8.2.4, and higher change the job_default_auto_cancel setting in $SPLUNK_HOME/etc/system/local/web.conf from the default value of 30 to 62.Details
|
| 2017-07-13 | SPL-143111 | "Splunkd daemon is not responding" when edit local windows event log collection |
Windows-specific issues
| Date filed | Issue number | Description |
|---|---|---|
| 2022-12-01 | SPL-233535, SPL-231086 | UF 9.x Unnecessary user creation during silent installation Workaround: Delete etc/passwd manually after installation, create user-seed.conf with new user in SPLUNK_HOME/etc/system/local and restart Splunk Customer claims that they were able to bypass the issue by using SPLUNKPASSWORD="" option during installation but I was not able to reproduce this
example of user-seed.conf:
https://drive.google.com/file/d/1GrypUL6719V0tGbw0Mv-u76jUYZ8LBS_/view?usp=sharing |
| 2022-11-16 | SPL-233007, SPL-234066 | KV Store fails to find the private key for a given certificate on Windows. It searches for -sslCertificateSelector subject=US Workaround: Use the splunkd generated default cert ($SPLUNK_HOME/etc/auth/server.pem) |
| 2022-03-19 | SPL-221019 | WEC + subscription with ContentFormat "Events" - indexed ForwardedEvents show "Splunk could not get the description for this event" for the "Message" field Workaround: Following steps should be followed: - to reconfigure subscription type to RenderedText: wecutil ss <subscription-name> /cf:RenderedText - in order to work around a MS defect on the WindowsEventViewer causing field description resolution failures within the WindowsEventViewer, when configuring RenderedText contentFormat you might want to also change the subscription locale, if not already done, to en-US: wecutil ss <subscription-name> /l:en-US and the same also for the datetime format on the WEC server to English (United States), see also here: https://serverfault.com/questions/606144/win2012r2-eventlog-subscription-dont-display-informations https://social.technet.microsoft.com/Forums/ie/en-US/3fd3d1fc-1194-4899-978c-3283085648bc/eventlog-forwarding-issues-either-the-component-that-raises-this-event-is-not-installed-on-your - please make sure to install the most recent Windows add-on compatible with your Splunk release, following the official installation documentation: https://docs.splunk.com/Documentation/AddOns/released/Windows/Install - please configure inputs.conf on the splunk instance running on the WEC server as follows, in order to onboard the ForwardedEvents data in XML format: [WinEventLog://ForwardedEvents] renderXml = true then save and restart splunk in order to apply the changes. - last, but not least, unless renderXml was set to true already before installing/upgrading to a regressed version, you will need to rewrite your searches and reports in order to comply with the new/XML-specific field extractions shipped in the Windows add-on, since the data is now onboarded in XML format. |
REST, Simple XML, and Advanced XML issues
| Date filed | Issue number | Description |
|---|---|---|
| 2020-07-28 | SPL-192792 | tsidxWritingLevel and other fields are set empty after updating index in UI |
| 2017-07-13 | SPL-143111 | "Splunkd daemon is not responding" when edit local windows event log collection |
| 2016-10-31 | SPL-131072 | Datamodel backend allows invalid time values |
PDF issues
| Date filed | Issue number | Description |
|---|---|---|
| 2016-11-23 | SPL-132925 | Table data rows generated with the addcoltotals command do not show up in PDF Workaround: If you are using addcoltotals to generate a totals data row, renaming the _time field can cause PDF generation issues.
Remove the label and |
Admin and CLI issues
| Date filed | Issue number | Description |
|---|---|---|
| 2024-04-26 | SPL-254998 | effective concurrency limit for scheduled searches in not updating in search prefs manager page |
| 2023-04-03 | SPL-238114 | messages.conf roles attribute not working as documented in messages.conf.spec |
| 2022-06-23 | SPL-226016, SPL-226271, SPL-229579 | Splunk crashed with SplunkConfigChangeWatcherThread if there is a symbolic link to a directory while config_change_tracker is enabled |
| 2022-06-21 | SPL-225949 | federated.conf.spec and federated.conf.example files are missing from the product build Workaround: Modify 'federated.conf.spec' and 'federated.conf.example' with the contents on the below document: https://docs.splunk.com/Documentation/Splunk/9.0.1/Admin/Federatedconf To check the workaround is working, restart the instance to see if logs appear. (See attached image: Image-20221110-234300.png) |
| 2021-03-26 | SPL-203060 | The splunkd process changes the local distsearch.conf on service start Workaround: There is no workaround. After upgrading to Splunk Enterprise 8.x, the splunkd process checks and modifies the local/distsearch.conf on each service start. The process will:
|
| 2020-07-28 | SPL-192792 | tsidxWritingLevel and other fields are set empty after updating index in UI |
| 2020-04-14 | SPL-186365 | Users are able to create/clone knowledge objects into apps where they lack permissions |
| 2019-08-05 | SPL-174406, SPL-109254 | Root unable to run splunk cli if SPLUNK_OS_USER is set |
| 2018-08-13 | SPL-158658 | A timeout or slow response when accessing Splunk Web Licensing page Workaround: A timeout or slow performance of the license management page is caused by a build-up of historical license warning messages, which are processed every time the page is accessed. Can be verified by running this search on the License Manager: | rest splunk_server=local /services/licenser/messages If a high value is returned for that end point, you are likely affected. Log a support ticket with Splunk to obtain a license reset key, and apply the key to clear out any historical license warning messages. After the reset license is applied, the license management pages should load normally. |
| 2017-11-29 | SPL-146820 | Unable to access some settings/manager pages (data model editor) if starting from the setup page of a non-visible app Workaround: Navigate to a visible app, such as the search and reporting app, and access the Splunk settings pages from that app context. |
| 2017-11-07 | SPL-146255 | limits.conf enable_clipping cloropleth setting is app/user tunable rather than global like the rest of limits.conf |
| 2017-04-03 | SPL-140747 | SSL connection in Python when using new ciphers may be slow. |
| 2016-11-09 | SPL-131880 | Reports/Alerts owned by the deleted user cannot be found in the Orphaned filter for the Reassign Knowledge Objects page |
Uncategorized issues
| Date filed | Issue number | Description |
|---|---|---|
| 2024-05-21 | SPL-256104 | Maximum daily volume for a pool displayed as Unlimited, when license maximum typed in manually in 'A specific amount' field Workaround: When setting up maximum daily volume for this pool, choose 'The license maximum' option. |
| 2024-03-13 | SPL-252573, SPL-251434 | Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment |
| 2024-03-13 | SPL-252571, SPL-251434 | Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment |
| 2024-03-13 | SPL-252572, SPL-251434 | Crashing Thread: typing_0 in Heavy Forwarder Workaround: apply to IHF/IUF/HF. etc/system/local/limits.conf [input_channels] max_inactive=300001 lowater_inactive=300000 inactive_eligibility_age_seconds=120 etc/system/local/inputs.conf [splunktcp-ssl:9996] queueSize=100MB note: ssl input port may be different on customer deployment |
| 2023-11-07 | SPL-246765, SPL-245974 | HTTP Event Collector s2s endpoint ignores all inputs.conf.spec. |
| 2023-11-07 | SPL-246766, SPL-245974 | HTTP Event Collector s2s endpoint ignores all inputs.conf.spec. |
| 2023-11-03 | SPL-246640 | web.conf server.socket_host no longer overrides splunk-launch.conf SPLUNK_BINDIP Workaround: No workaround available. |
| 2023-05-02 | SPL-240700 | A forwarder that does not have a 'clientCert' set in its outputs.conf configuration file will not connect over TLS to a receiver even when the receiver has 'requireClientCert' set to "false" in its inputs.conf configuration file Workaround: Specify a value for 'clientCert' in the outputs.conf file on the forwarder. Ensure that the file you specify for the setting exists and is a valid file in Privacy Enhanced Mail (PEM) format. |
| 2023-01-06 | SPL-234643 | Splunkd abort - due to 3rd party S2S client unable to process ACKs. Workaround: For some versions of 3rd-party S2S client, there is an option to change the behavior of a failed ACK. For example, turning off "Minimize in-flight data loss". |
| 2022-12-09 | SPL-233858 | Splunk kernel drivers have expired on existing Windows installations |
| 2022-10-05 | SPL-231139, SPL-228404 | Following upgrade to 9.0 customer HF blocks queues when Ingestion latency bug hits Workaround: disable what we can from ingestion latency reporter in health.conf on HF: \[health_reporter] aggregate_ingestion_latency_health = \[0|1]
Disable that please. And the feature itself. \[feature:ingestion_latency] alert.disabled = 1 disabled = 1 |
| 2022-07-29 | SPL-227621, SPL-241402, SPL-227998 | Export of search results in the GUI fails with "Service Unavailable" with PYTHONHTTPSVERIFY=1 Workaround: Turn off PYTHONHTTPSVERIFY for now |
| 2022-07-28 | SPL-227579, SPL-226751 | SSG Modular Inputs Stuck in Enable-Disable Loop |
| 2022-07-21 | SPL-227153 | After upgrade to 9.0, external indexes are missing from searchable choice list when creating a new role Workaround: The following setting needs to be enabled in server.conf: [introspection:distributed-indexes]
https://docs.splunk.com/Documentation/Splunk/9.0.0/Admin/Serverconf |
| 2022-07-14 | SPL-226855 | modify server roles check on splunk-assist |
| 2022-06-30 | SPL-226400, SPL-226485 | Queues blocked infinitely with useACK and autoBatch. Workaround: Turn off useACK useACK=false |
| 2022-05-25 | SPL-225455 | Splunk Assist: On indexer cluster managers, an "Error loading assist: try the operation again or contact Splunk support" message appears |
| 2022-04-12 | SPL-222543, SPL-224946 | Unable to generate diag - "UnicodeDecodeError: 'utf-8' codec can't decode byte XxXX in position YY: invalid start byte" Workaround: The problem is caused by non-ASCII/UTF-8 characters, that are present in your configuration and are not supported. You can remediate the problem: 1. Either remove non-ASCII/UTF-8 characters from your configuration files.
2. Or take a backup of '$SPLUNK_HOME/lib/python3.7/site-packages/splunk/clilib/cli_common.py' and in line 127:
Add parameter to "line.decode" - either "errors='replace'", or "errors='ignore'". Eg:
line.decode(errors='replace') |
| 2022-04-06 | SPL-222105 | When all inherited roles are taken out from admin role, it will cause admin user failed to show other users even though all capabilities is set natively. Workaround: Two possible approaches: 1. Remove the option grantableRoles = admin from authorize.conf - this is not permanent workaround and will need to be done every time admin role is modified. 2. Add any capabilities that the other user roles have to the "admin" role. |
| 2022-02-24 | SPL-219715, SPL-225376, SPL-225374, SPL-225375 | Workload Management fails to enable on restart if a rule contains a role that is missing on the platform |
| 2021-04-24 | SPL-204740, SPL-204735 | Deletion of a workload pool is allowed if there is a 'disabled' rule that is related to that workload pool and this can cause errors if the rule is re-enabled later Workaround: To prevent this issue: When you delete a workload pool, please make sure that you delete any disabled workload rules that are associated with that workload pool. To resolve the issue if you encounter this: Disable or delete the workload rule that is associated with a workload pool that does not exist anymore. |
| 2021-04-20 | SPL-204428, SPL-203620 | AWS SDK log messages should not be turned on for on-prem builds Workaround: add category.AwsSDK=FATAL under the [splunkd] stanza in log-local.cfg |
| 2021-03-19 | SPL-202682 | The license usage report tab name is Previous 60 days, but the reports run over the last 30 days |
| 2021-02-10 | SPL-200532 | SmartStore: Stuck fixup due to inability to freeze unsearchable/unstable bucket Workaround: This issue is caused by a single unsearchable bucket that has been frozen while not existing on remote storage. The bucket copy on the peer node's cache remains stuck in the fixup state, resulting in messages to the effect that all data is not searchable, the replication factor is not met, and the search factor is not met. To resolve, on the peer node, invoke the "/services/cluster/peer/buckets" endpoint, specifying the faulty bucket, setting "search_state=Searchable" to make the bucket searchable. You do not need to restart the peer node afterwards. Here is the syntax for the required endpoint: curl -k -u admin https://<peer_node_with_bucket>:<mgmt_port>/services/cluster/peer/buckets/<bucket_id>/change_bucket -d bucket_mask=0 -d search_state=Searchable -d generation_id=0 -d searchable_sources="peer,site,server_name,host_port_pair,replication_port,replication_use_ssl,searchable,bucket_mask" Note that pairs of angle brackets indicate variables that must correspond to your instance and bucket. |
| 2020-10-01 | SPL-195810 | Using CLI command to stop migration of KVstore on a SHC running on Windows OS can cause the SHC captain to reach an invalid state Workaround: Restart the SHC captain |
| 2020-08-10 | SPL-193389 | Parallel upload is not supported in gcp-sse-kms encryption mode Workaround: In the volumes using gcp-sse-kms encryption mode, specify "remote.gs.upload_chunk_size = 0" to disable parallel upload. |
| 2020-07-30 | SPL-192936 | Subsecond search - When you update metric.timestampResolution via the UI, it is not updated on the search head index.conf file. This does not affect search functionality. |
| 2020-05-06 | SPL-188800 | Starting Splunk software with incorrect KV store storage engine causes KV store to crash Workaround: In the [kvstore] stanza of your server.conf file, set the storageEngine setting to match the storage engine that you're using, either wiredTiger or mmapv1. To learn which storage engine you're using, check whether the file extensions in the var/lib/splunk/kvstore/mongo directory are *.wt for Wired Tiger or *.ns for Memory Mapped. |
| 2019-10-03 | SPL-177447 | Bundle replication takes longer than expected time for indexers that have bundleEnforcerBlacklist configured |
| 2019-09-26 | SPL-177144, SPL-177326 | Under heavy search workload, the search memory usage estimation may be higher than actual usage |
| 2019-09-25 | SPL-177008, SPL-176710, SPL-177009 | Workload management fails to enable for addition of a pool with 1% cpu and 1% memory |
| 2019-09-16 | SPL-176514 | Offline rebuild of unsearchable bucket may lead to stale information in dbinspect searches |
| 2019-09-13 | SPL-176447 | SmartStore: Migration uploads of auto_high_volume buckets can fail indefinitely due to an XFS bug Workaround: Before migration, lower the max_concurrent_uploads setting in server.conf to 2. After migration, revert the setting to the default of 8. |
| 2019-07-19 | SPL-173449, SPL-173259 | timezone isn't stored for start_time/end_time of rule schedule every_day/every_week/every_month |
| 2019-03-26 | SPL-168314 | SmartStore standalone instance + Monitoring Console: Bootstrapping panel needs to reflect the standalone bootstrapping process |
| 2018-03-20 | SPL-152330, SPL-151992 | After installing Splunk on Windows using msiexec and the "GENRANDOMPASSWORD=1" option (and if generated password ends with backslash) admin is unable to login with msg "No users exist. Please set up a new user." Workaround: Create a $SPLUNK_HOME/etc/system/local/user-seed.conf and restart Splunk [user_info] |
| 2017-06-29 | SPL-142789, SPL-95144 | Indexed message for Windows security event logs shows "FormatMessage error" Workaround: Splunk believes this was introduced in a Microsoft Windows patch. The workaround is to configure a delayed start of the Splunk service(s) so that it starts after the Windows Event Log service. |
| 2017-05-09 | SPL-141693 | DataModel Editor - when child object has same name as inherited field, inherited field does not show in the inherited fields list. |
| 2017-01-06 | SPL-134707 | Splunk restart does not create missing server.pem certificate on Windows Workaround: Use bin/splunk createssl server-cert -d etc/auth/ -n server to generate a new certificate. |
| 2016-11-21 | SPL-132670 | Mac OS 10.11: disable boot-start doesn't remove the file /Library/LaunchAgents//com.splunk.plist by enabling boot-start in prior Splunk/UF |
| 2016-08-31 | SPL-127800 | Opting in to data sharing on a monitoring console produces duplicate data |
| 2016-07-26 | SPL-125052 | Sole Admin can demote themself to Power without path of recovery in GUI. Workaround: Through the command line, you can open notepad and modify the password file to regain 'Admin' status. |
| 2016-06-21 | SPL-123174 | JSON indexed_extractions doesn't work for TCP inputs |
Splunk Analytics for Hadoop
| Date filed | Issue number | Description |
|---|---|---|
| 2017-04-04 | ERP-2040 | Splunk archiving fails for large block sizes (buckets) due to HDFS write crashes for Hadoop version 2.8, 2.7.x Workaround: Upgrade Hadoop to 2.8.2 or higher. |
| 2015-09-09 | ERP-1650 | timestamp data type not properly deserialized. |
| 2015-08-05 | ERP-1619 | Searching on a newly created archive index before the bucket copy saved search is run causes a filenotfound exception. Workaround: Reenable the bucket copy saved search and let it run, or force the archiving to happen via | archivebuckets force=1 and then rerun the search. |
| 2015-07-07 | ERP-1598 | minsplit rampup - splits generation takes too long. Workaround: Set minsplits=maxsplits |
| 2015-05-12 | ERP-1502 | Non-accelerated pivot search on Pivot UI page waits for a long time to return result. |
| 2015-01-08 | ERP-1343, SPL-95174 | Splunk Analytics for Hadoop searches fail on corrupted journal.gz files, although Splunk searches run without error. Workaround: Add the journal.gz to the input path's blacklist (vix.input.1.ignore = ....) |
| 2014-10-27 | ERP-1216 | Data Explorer preview does not honor existing sourcetypes for big5/sjis files. |
| 2014-10-03 | ERP-1164 | Report acceleration summary gets deleted when two Splunk Analytics for Hadoop instances point to the same Splunk working directory. Workaround: To mitigate this issue, make sure that vix.splunk.home.hdfs (or Working directory in the UI) is unique on both search heads that are not in a pool. To keep your instances in the same working directory, configure vix.splunk.search.cache.path to be unique on both search heads. |
| Welcome to Splunk Enterprise 9.0 | Increased skipped search rate after upgrade to 9.0 |
This documentation applies to the following versions of Splunk® Enterprise: 9.0.0
Download manual
Feedback submitted, thanks!