All DSP releases prior to DSP 1.4.0 use Gravity, a Kubernetes orchestrator, which has been announced end-of-life. We have replaced Gravity with an alternative component in DSP 1.4.0. Therefore, we will no longer provide support for versions of DSP prior to DSP 1.4.0 after July 1, 2023. We advise all of our customers to upgrade to DSP 1.4.0 in order to continue to receive full product support from Splunk.
Troubleshoot lookups to the Splunk Enterprise KV Store
Use this page to troubleshoot common issues with lookup connections to the Splunk Enterprise KV Store.
You are experiencing latency or performance issues with a KV Store lookup
If you are experiencing performance or latency issues in an active pipeline with a connection to a Splunk Enterprise KV Store, make sure you are sizing your Splunk Enterprise environment appropriately.
Cause: You do not have an appropriately sized distributed Splunk Enterprise environment
If you want to connect the to a Splunk Enterprise KV Store on a distributed Splunk Enterprise environment, you must make sure that your Splunk Enterprise environment is sized appropriately. The Splunk Enterprise KV store can support approximately 45,000 requests per second per search head cluster node. For example, a search head cluster with three nodes can handle approximately 135,000 requests per second. To perform lookups to a Splunk Enterprise KV Store, the makes repeated requests to the Splunk Enterprise KV Store. If you do not have an appropriately sized distributed Splunk Enterprise environment, your DSP pipeline might receive data at a higher rate than it can process, resulting in backpressure. Therefore, best practices are to scale your Splunk Enterprise search head cluster appropriately to handle your peak pipeline throughput.
Use the following steps as a reference on how to calculate how many search head cluster nodes you need. These steps assume that you have already created a connection to a Splunk Enterprise KV Store and are using that connection in an active pipeline.
- From the UI, open the active pipeline containing your KV Store lookup.
- Find the lookup function in your pipeline and copy the
Events Per Second
(EPS) number to a preferred location. - Estimate the cache miss rate of your lookup connection. For assistance with this, contact Splunk Support.
- Get the
batch_size
of your KV Store lookup.- Log in to the Splunk Cloud Services CLI.
./scloud login
- Get details about your connections. Locate the KV Store connection from the returned list and copy the
batch_size
value to a preferred location. If you do not see thebatch_size
value, then your connection uses the defaultbatch_size
of 1000../scloud streams list-connections
- Log in to the Splunk Cloud Services CLI.
Now that you have the Events Per Second
, the batch_size
, and Cache Miss Rate
you can calculate approximately how many search head cluster nodes you need using the following formula.
( Events Per Second / Batch Size) * Cache Miss Rate = Requests per second to the KV Store
As an example, assume that you have the following:
- A lookup function processing 8,000,000 Events Per Second (EPS).
- A lookup batch size of 100 records.
- A cache miss rate of 70%.
Using the formula above, your pipeline is sending (8,000,000 / 100) * .7 = 56,000 requests per second to the Splunk Enterprise KV Store. Since each search head cluster node can handle approximately 45,000 requests per second, a request load of 56,000 requests per second would require a Splunk Enterprise cluster that contains at least 2 nodes. You can also reduce the request load by increasing the batch_size
or the cache_size
of the KV store lookup. See the Connect to the Splunk Enterprise KV Store using the Streams API section for more information on these two settings, and see About lookup cache quotas for more information about lookup cache sizes.
About lookup cache quotas | Create custom functions with the SDK |
This documentation applies to the following versions of Splunk® Data Stream Processor: 1.2.1, 1.2.2-patch02, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5
Feedback submitted, thanks!