Known issues for DSP

This version of the Splunk Data Stream Processor has the following known issues and workarounds.

If no issues appear here, no issues have yet been reported.

Date filed	Issue number	Description
2022-11-21	DSPOP-7363	Postgres-backup uses incorrect Postgres version
2022-11-16	DSPOP-7296	Upgrade from 1.2.x to 1.3.x sometimes fail on prometheus-writer deployment Workaround: <DISABLE REMOTE WRITER> ./set-config K8S_SPLUNK_ENTERPRISE_METRICS_ENABLED false ./deploy <RUN UPGRADE> <ENABLE REMOTE WRITER> ./set-config K8S_SPLUNK_ENTERPRISE_METRICS_ENABLED true ./deploy
2022-05-23	DSPOP-5588	./set-config (and other Bash scripts) fail when passing in values with whitespace characters (multiple single-word arguments separated by whitespace are fine). ./set-config can also fail when values start with dashes. Workaround: For the whitespace issue - in general for a given Bash script, it can be edited to add quotes around the $@ to preserve quotes passed in the arguments (so "$@") to the DSP CLI. ./set-config is the most likely to have values that have whitespace. Alternatively, the DSP CLI can instead be used directly, like dsp config set <key> "<value>". For the - issue with ./set-config - this is a limitation of CLI parsing rather than a bug. This issue can be worked around by either using a = (<key>=<value>), or adding -- before the key-value pair(s). Examples (after editing set-config to use "$@", with quotes): ./set-config K8S_PULSAR_BOOKIE_MEM_OPTS=" -Xmx3G -XX:MaxDirectMemorySize=3G" ./set-config -- K8S_PULSAR_BOOKIE_MEM_OPTS " -Xmx3G -XX:MaxDirectMemorySize=3G"
2022-05-18	DSPOP-5566	HEC Async client stops processing data when error occurs Workaround: Update pipeline by setting the following property and restarting the pipeline: http-async = false This change will cause the pipeline to restart on any error with the HEC Sync, but data will continue to flow when the pipeline restart.
2022-05-10	DSPOP-5493	Upgrade from 1.2.x to 1.3.0 fails if DSP was installed with a custom state directory (--location or --state-dir)
2021-12-23	DSPOP-4204	Upgrade fails when root's shell is set to nologin
2021-10-05	DSPOP-4205	Pulsar connector: When you create a connection to Apache Pulsar, the server_ca_certificate argument is labeled as optional but is required. If you leave this field blank, pipeline activation fails with error "Failed to fetch". Workaround: Edit the Pulsar connection and upload the original CA certificate.
2021-10-04	DSPOP-4206	Centos: The sudo ./scloud login command does not work on Centos operating systems. Workaround: Log in to SCloud without sudo: ./scloud login.
2021-05-27	DSPOP-4207	While upgrading, the upgrade process is killed and ./gravity status returns collecting_garbage status for your cluster. Workaround: Run following command on DSP master console: ./sudo gravity plan complete
2021-05-26	DSPOP-5293	Multi node clusters do not work on certain RHEL versions running on VMWare Workaround: A regression introduced into the RHEL 8.3 linux kernel prevents multi node DSP clusters from working properly on VMWare infrastructure when the DSP VMs are scheduled across multiple ESX hosts. This regression also exists in RHEL 8.6. To workaround, disable checksum offloading for impacted DSP network interfaces. ethtool -K cni0 tx off ethtool -K flannel.1 tx off Note: this must be run on every node in the cluster and every system boot. The system should recover within a few minutes. More information about the nature of the issue is found at https://access.redhat.com/solutions/5881451
2021-04-27	DSPOP-4208	When you try to send data to an Amazon S3 bucket that has a period (.) in its name, the pipeline fails to send the data and restarts indefinitely.
2021-04-09	DSPOP-4209	When you modify a Union function using the UI Form, the pipeline canvas does not automatically update to show your changes. Workaround: Refresh the pipeline canvas by doing any of the following: save the pipeline, add a function, delete a function, or rename a function in the pipeline.
2021-02-02	DSPOP-4219	Upgrades time out due to pods in ContainerCreating status. Workaround: 1. Restart the Kubelet service by running the following command: gravity exec systemctl restart kube-kubelet 2. Resume the upgrade: gravity plan resume 3. Wait for the upgrade to complete: gravity plan --tail 4. Run the post-upgrade steps: ./post-upgrade
2020-10-12	DSPOP-4210	When editing an active pipeline, toggling between builders reverts the pipeline definition to the last activated version. Workaround: When editing an active pipeline, stay in the Canvas builder or clone the active pipeline and make any desired changes to the cloned pipeline.
2020-10-05	DSPOP-4481	Large batch size when using a clustered KV Store behind a GCP load balancer causes HTTPs requests to exceed allowed size (status code 431). Workaround: Reduce the batch size limit to make the request under 16KB.
2020-09-11	DSPOP-4211	The Write Thru KV Store function configuration cannot be updated through the Edit Streams JSON option in the pipeline metadata. Workaround: Use the View Configurations tab to update the function configuration.
2020-08-17	DSPOP-4463	DSP does not run on an operating system that has FIPS mode enabled.
2020-07-16	DSPOP-4212	Forwarders routing through DSP will not appear in the Monitoring Console Forwarder Deployment dashboards.
2020-05-06	DSPOP-4213	Custom UI for the Select function does not work when the incoming data schema contains a field name that requires single quotes. Workaround: 1. Click on the SPL expressions table at the top right corner of the select function's configuration panel. 2. Input the field names with/without quotes into the editor box as valid SPL statements like 'offset' as kafka_offset. 3. Multiple entries could be added by the add button below.
2020-04-21	DSPOP-4214	You cannot switch between the Canvas and the SPL Builder if the functions are not configured correctly. Workaround: Configure every function so that the UPL is valid.
2020-04-01	DSPOP-4215	Some services fail to run on Ubuntu 16.04 with old kernels. Workaround: 1. Update your Ubuntu kernel packages to the most recent available. Run this command on each node in the cluster: apt update && apt upgrade -y && apt dist-upgrade -y 2. Reboot each node in the cluster to ensure the new kernel is running.
2020-03-27	DSPOP-4216	Stopping and starting an S3 sink pipeline leaves behind unfinalized _part files.
2020-02-28	DSPOP-4217, DSP-18225	Users can remove their own admin privileges on the User Management page.