Splunk App for Data Science and Deep Learning Search History and results caching

Efficient management and retrieval of search results is crucial for reducing system load and ensuring timely access to valuable insights, especially when working with compute-intensive algorithms like large language models (LLMs).

The Splunk App for Data Science and Deep Learning (DSDL) version 5.2.1 introduces a specialized Search History feature that leverages the Splunk platform's built-in summary indexing to persist and reuse expensive search results.

Benefits of Search History

Review the following benefits of the Search History feature:

Benefit	Description
Persist valuable outputs	By default, search results in the DSDL are not stored. If a result is needed again, the search must be rerun. Using Search History ensures that these results are saved for future access.
Performance and efficiency	Storing results in a summary index means you can quickly retrieve previously processed data, reducing the need to repeat resource-heavy computations. This is especially beneficial for algorithms that return lengthy text outputs or when auditing past analyses.
Reuses standard Splunk platform features	By using the Splunk platform built-in summary index capabilities, the feature ensures compatibility, reliability, and security for your stored search data.

How to turn on Search History

Complete the following steps:

1. Turn on Essential Saved Searches in your Splunk platform instance. These settings are stored in the saved searches settings:
   • get_search_jobs_save_to_summary
   • get_search_results_save_to_summary
2. Select algorithms for history retention:
1. Edit the search_history_enabled_algos.csv lookup table containing both algo_name and algo_enabled.
2. Find the name of the algorithm you want to enable and set its value in the algo_enabled column to 1.

   By default, only LLM algorithms have history enabled, as they often involve lengthy processing times and generate extensive text outputs.

Turning on history for algorithms that output large numeric tables might lead to significant performance or storage concerns. Consider selectively enabling this feature only where high resource consumption or frequent result reuse is expected.

What happens after Search History is turned on?

The following changes are in place after you turn on Search History:

• Results for selected algorithms are stored in summary indexes.
• Users can retrieve these results instantly for quick analysis or auditing, without the need to rerun the original, potentially expensive, search.

See also

For more information on how Splunk platform summary indexing can accelerate searches and enhance data management efficiency, see Use summary indexing for increased search efficiency.