Review the Workload (preview) dashboard
Better understand your usage and license entitlement metrics in the Cloud Monitoring Console (CMC) version 3.25.0 Workload (preview) dashboard.
Preview features are provided by Splunk to you "as is" without any warranties, maintenance and support, or service level commitments. Splunk makes this preview feature available in its sole discretion and may discontinue it at any time. Use of preview features is subject to the Splunk General Terms.
What is Splunk Virtual Compute (SVC)?
SVC is a unit of capabilities that includes CPU, memory, and I/O. Overall SVC usage primarily considers CPU across search and indexing workloads. Splunk deploys infrastructure based on your entitled SVCs.
For more information about the SVC entitlement for your workload-based subscription, see Performance considerations in the Splunk Cloud Platform Service Description. Be sure to view the correct version for your Splunk Cloud Platform deployment version.
If your organization has an ingest-based subscription, the SVC license entitlement metric and suggested thresholds are not applicable to your deployment. To determine the appropriate SVC entitlement for your deployment and to convert your ingest-based subscription to a workload-based subscription, contact your Splunk account representative.
Access the Workload (preview) dashboard
To investigate your usage metrics over a specific time range and identify where you can optimize your organization's SVC consumption, use the Workload (preview) dashboard in CMC:
- In Splunk Web, select Cloud Monitoring Console.
- Select License Usage and then Workload
- On the banner, select the Workload (preview) dashboard link.
Review the top-level panels
The top-level panels display your deployment's peak usage indicators. Use these panels to gauge your deployment's peak SVC usage during a given time interval. Select the question mark icon for more information or see the following table to learn more about each indicator:
Panel | Description |
---|---|
Current license entitlement |
Shows the number of SVCs assigned to your organization's subscription for your license entitlement. |
Overall • Peak SVC usage |
Shows your organization's overall peak SVC usage as a single value and a percentage of your license entitlement. Splunk deploys infrastructure based on your entitled SVCs. Overall peak SVC usage refers to the highest amount of resources used in a given time interval to perform system processes such as indexing, any running search processes, and shared services. It primarily measures the CPU usage across search and indexing workloads. Generally, ensure SVC usage is less than 80% to maintain performance. Usage greater than or equal to 80% is considered elevated, and greater than or equal to 90% might cause degraded performance. |
Search • Peak SVC usage |
Shows your organization's search workload peak SVC usage as a single value. Search peak SVC usage refers to the highest amount of resources used in a given time interval to perform search processes. It primarily measures the CPU usage across search workloads. The search workload can occur on both the search and indexing tiers. |
Indexing • Peak SVC usage |
Shows your organization's indexing workload peak SVC usage as a single value Indexing peak SVC usage refers to the highest amount of resources used in a given time interval to perform indexing processes. It primarily measures the CPU usage across indexing workloads. The indexing workload occurs on the indexing tiers. |
Indexer memory utilization |
Shows the 90th percentile measurement of the memory used by all processes running across the time frame selected for all the indexer hosts. The 90th percentile measurement captures 90% of the values below the estimate. 10% of the values above the estimate are excluded as outliers. |
Indexer cache churn |
Shows the percentage of cache churn for your stack. Indexer cache churn is the rate at which data is evicted from local disk cache to make room for new data. Cache churn occurs when the cache is unable to retain frequently accessed data due to capacity constraints or inefficient cache management, resulting in data being replaced more frequently than desired. Repeatedly evicted data needs to be reloaded from slower storage, which can lead to performance degradation, increased search latency, and inefficient caching. High cache churn is often an indication of inefficient searches or a need for more capacity, particularly in environments with high data volumes or complex search patterns. For tips on how to improve your cache churn percentage, see Optimize indexing and search processes |
Indexer CPU utilization |
Shows the 90th percentile measurement of the CPU used by all processes running across all indexers. The 90th percentile measurement captures 90% of the values below the estimate. 10% of the values above the estimate are excluded as outliers. |
Review the overall, search, and indexing workload panels
The next panel displays further information about your overall, search, and indexing workloads. Select the respective panel tabs to view detailed charts on specific processes. Select each workload to view its metrics.
Overall workload panel
This Overall workload • Peak SVC usage panel shows your organization's SVC usage in the context of your license entitlement.
Select from the following views:
- Overall: The highest amount of resources used in a given time interval to perform system processes such as indexing, any running search processes, and shared services.
- By process: Overall peak SVC usage split by search processes, indexing processes, and shared services.
- By tier: Peak SVC usage based on processes performed by the search head and indexing tiers.
The Top 10 apps chart shows apps that contribute to the highest search time or estimated SVC usage.
The Top 10 users chart shows users that contribute to searches with the highest search time or estimated SVC usage. These users may be human or virtual administrators.
The internal splunk-system-user
virtual administrator runs jobs and processes like summary refreshes, report accelerations, and data model accelerations on behalf of a Splunk Cloud Platform customer. Running these processes consumes SVCs. If the SVC usage of splunk-system-user
seems abnormal, contact the deployment's administrator to investigate the increased consumption.
Search workload panel
The Search • SVC usage panel displays search processes that occur on the search and indexing tiers. The sum of these processes equals the peak SVC usage from search processes during this time interval.
Select from the View by options to view estimated SVC usage or search time in seconds.
Select from the following Search head options:
Search head | Description |
---|---|
All | Shows all search heads in your Splunk Cloud Platform deployment. This category includes all the data ingested and processed in the deployment. |
Specific search head name | Shows data for a specific search head that is ingested, processed, and summarized in the CMC 2.9.0 and higher. |
Select from the following Split by options:
Search head | Description |
---|---|
Apps | Lists a maximum of the top 10 apps and their respective search workload SVC consumption or search time. |
Searches | Shows which searches use the most search workload SVC or search time as a percentage of the total consumption. |
Search type | Shows search types and their search time or estimated SVC consumption. |
Users | Lists a maximum of the top 10 users and their search workload SVC consumption or search time. These users can be human or virtual administrators. |
The Search • SVC usage tracks the following search types:
Search type | Description |
---|---|
REST_API | Searches that use the Splunk REST API. See Basic concepts about the Splunk platform REST API in the Splunk Enterprise REST API user manual. |
ad-hoc | Searches that are unscheduled and manually run. See ad hoc search. |
dashboard | Searches run by your dashboards |
scheduled | Searches that are saved and scheduled so they automatically run. See scheduled search. |
scheduled realtime | Searches where the search_mode field value is realtime indexes RT Indexes for realtime indexes and the search_type field value is scheduled .
|
summary director | Maintenance tasks that run in the background involving caching and summarization to ensure searches are processed. |
report acceleration | Searches that are related to accelerated data models or reports. See data model acceleration, report acceleration, and How data model acceleration differs from report acceleration and summary indexing in the Splunk Enterprise Knowledge Manger Manual. |
Other | Uncategorized usage. |
The Dispatched and skipped search count per hour chart shows the number of searches per hour that are dispatched or skipped.
Indexing workload panel
Indexing workload • usage per hour panel encompasses ingestion and indexing processes on the indexing tier. The sum of these processes equals the peak SVC usage from indexing processes during this time interval.
Select from the Split by options to view indexing processes by specific indexes or source types.
The Ingestion by hour chart shows hourly rate of ingestion. When data ingestion rates are high, the indexer consumes more resources to process and ingest data. High ingestion rates can increase SVC usage.
Interpret Workload (preview) dashboard metrics
SVC utilization is not a direct measure of your deployment health. To better understand your deployment, go to the Health dashboard and see Use the Health dashboard.
You can turn on preconfigured alerts about your workload and SVC utilization with the Alerts dashboard. See Use the Alerts dashboard to learn more.
Optimizing search and indexing processes can improve SVC utilization and might improve system performance. To learn more, see Optimize indexing and search processes.
Use the License Usage dashboards | Use the Forwarder dashboards |
This documentation applies to the following versions of Splunk Cloud Platform™: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403, 9.3.2408
Feedback submitted, thanks!