Splunk Cloud

Splunk Cloud User Manual

Download manual as PDF

Download topic as PDF

Monitor Splunk Cloud deployment health

The Cloud Monitoring Console lets Splunk Cloud administrators view information about the status of your Splunk Cloud deployment. Cloud Monitoring Console dashboards provide insight into how the following areas of your Splunk Cloud deployment are performing:

  • search
  • indexing
  • forwarder connections
  • indexer clustering and search head clustering, if applicable
  • license usage, if applicable.

Locate the Cloud Monitoring Console

If you have a self-service Splunk Cloud deployment, to find the Cloud Monitoring Console:

  1. Click Settings.
  2. Click the Cloud Monitoring Console icon on the left.

If you have a managed Splunk Cloud deployment, the Cloud Monitoring Console is an app.

  1. From anywhere in Splunk Web, click Apps.
  2. Click Cloud Monitoring Console.

On the App Management page, the Cloud Monitoring Console is named splunk_instance_monitoring.

Dashboards

The Splunk Cloud Monitoring Console app provides information about your Splunk Cloud performance. The information is organized into several dashboards.

Dashboard Description For more information
Overview Information about the performance of your Splunk Cloud deployment, including license usage (if applicable), indexing performance, and search performance information. About license violations in the Splunk Enterprise Admin Manual
What Splunk Cloud does with your data in Getting Data In

About jobs and job management

Search Usage Statistics Information about how your users are running searches. Write better searches in the Splunk Enterprise Search Manual
Configure the priority of scheduled reports in the Reporting Manual
Scheduler Activity Information about how search jobs (reports) are scheduled. Configure the priority of scheduled reports in the Reporting Manual
Skipped Searches Information regarding skipped searches and search errors. This dashboard is available on managed Splunk Cloud deployments. See Prioritize concurrently scheduled reports in Splunk Web and Offset scheduled search start times in the Reporting Manual, and Troubleshoot high memory usage in the Splunk Enterprise Troubleshooting Manual.
Indexing Overview Incoming data consumption. Look for an indexing rate lower than expected (for example, an indexing rate of 0 with a forwarder outgoing rate of 100). Filter by source type to discover source types that are sending a larger volume than expected. This dashboard is available on managed Splunk Cloud deployments.
User Activity Statistical information about users, page views, and apps. Note if a particular user has an large number of page views. This might indicate that users are sharing credentials, or that the account is being used on a dashboard with multiple refreshes, creating additional search load. This dashboard is available on managed Splunk Cloud deployments.
HTTP Event Collector Status of HTTP event collection, if you have enabled this feature. Set up and use HTTP Event Collector in Getting Data In
Data Quality Displays any line breaking, aggregation, or event breaking errors in your incoming data. Resolve data quality issues in Getting Data In
Upgrade Readiness Provides information about whether your Splunk Cloud instance is ready for upgrade, including analysis of your apps and forwarders. Additional information is provided for specific upgrade paths. Verify your Splunk Cloud instance is ready to upgrade
Forwarders: Instance and Forwarders: Deployment Information about forwarder connections and status. To get data to appear on the two forwarder dashboards, navigate to Monitoring Console > Settings > Forwarder Monitoring Setup, or click the setup link in either of the forwarder dashboards and follow the setup instructions. Troubleshoot forwarder/receiver connection in Forwarding Data
Infrastructure-based Licensing dashboard Information about your organization's Splunk Virtual Core (SVC) license limit and usage so you can take corrective action and conserve SVCs. To view this dashboard, navigate to Cloud Monitoring Console > License Usage > Infrastructure-based Licensing. Monitor SVC usage with the Infrastructure-based Licensing dashboard


Check your total data retention capacity

When you send data to Splunk Cloud, it is stored in indexes, and you can self-manage your Splunk Cloud index settings using the Indexes page in Splunk Web. Splunk Cloud retains data based on index settings that enable you to specify when data is to be deleted. Data retention capacity space in your Splunk Cloud service is based on the volume of uncompressed data that you want to index on a daily basis. Splunk Cloud provides 90 days worth of data retention capacity with every subscription. For example, if your daily volume of uncompressed data is 100 GB, your Splunk Cloud environment will have 9000 GB (9 TB) of data retention capacity. You can also purchase additional data retention capacity.

The Cloud Monitoring Console (CMC) Indexes and Storage dashboard provides insights into your data use so that you can better understand your current usage and predict future licensing needs.

In the Indexes and Storage dashboard, the CMC provides insights into your data retention based on the uncompressed data you have indexed.

Steps to find your retention usage and set an alert

  1. Go to CMC > Indexes and Storage.


  2. The graphic shows the Indexes and Storage page of the Cloud Monitoring Console. It is intended to orient the user.

  3. Take note of your total index size, which displays in the upper right. This represents your total uncompressed data that is currently retained.


  4. The graphic shows the total size of the uncompressed data for the index. The image is intended to help the user locate this field.

  5. Compare this value to your licensed entitlement amount to see if you need to update your license based on current usage. If you do not know your licensed entitlement, reach out to your Splunk sales representative.


  6. Finally, create a query against CMC, and configure Splunk Cloud to generate an alert if the value exceeds your licensed usage. The following sample query shows the alert where license_gb=10000000 should be replaced with your licensed data ingestion value (in GB):


  7.  | dbinspect index=* cached=t
    | where NOT match(index, "^_")
    | stats max(rawSize) AS raw_size BY bucketId, index
    | stats sum(raw_size) AS raw_size
    | eval raw_size_gb = round(raw_size / 1024 / 1024 / 1024 , 2), license_gb = 10000000, storage_usage_pct = round(raw_size_gb / license_gb * 100, 2)
    | fields storage_usage_pct
    


  8. Note that the query should be run against All Time.


More information

For more information about creating indexes, see Manage Indexes. For detailed instructions on creating alerts, see Alerts.

Understand your index data retention capacity

Your licensed data retention capacity is based on two variables: the daily licensed ingestion rate (e.g. 1 TB per day) and the amount of time Splunk Cloud is licensed to retain your data (e.g. 30 days). To understand how your data retention compares to your licensed retention, it's a good idea to view details about your index storage.

When you configure data retention for an index, you also configure two variables: the size of the index, and the number of days to retain the data. For example, you set data retention for 10 TB or 90 days, whichever comes first. If your data is retained for less time than you configured, it's likely that your ingestion rate is higher than expected. For example, if you configured your index to store data for 90 days or 10 TB, and you see that the data is being retained for 10 days, it's likely that you have hit the 10 TB threshold much sooner than expected, indicating a high ingestion rate. On the other hand, a longer retention than expected could indicate a misconfiguration of your index settings (i.e., you configured data retention for a time period that exceeds your licensed retention).

Steps to investigate your indexes

Steps Detailed Image
Go to CMC > Indexes and Storage. The graphic shows the Indexes and Storage page of the Cloud Monitoring Console. It is intended to orient the user.
A good method to determine if your data usage is running higher than expected is to check the date of the first event and the date of the last event and compare it to the retention setting for the individual index. For example, if the first event is 12/13/17, and the last event is 12/23/17 and the retention setting for the index is 90 days, then the data ingestion for the index was met long before the time retention setting was met. So, the data ingestion was greater than anticipated.


The graphic shows a detail of the First and Last event in the Cloud Monitoring Console. It is intended to display how to use the events to understand if data usage estimates are accurate.

Next, check to see which indexes are larger than others. You want to find which index is consuming the most storage and why.
  • To do this, check the index size, which shows the uncompressed data retained by the index.
  • Click the Index Size (GB) heading to sort the indexes by size.
  • Click the name of a larger index to open the Index Details page.
The graphic shows a detail of the Index size in the Cloud Monitoring Console. It is intended to help the user locate index size information.
In the Index Details page, you can see if there's a spike or a higher trend line for an index. Both of these data points are clues that will tell you that you may need to adjust index settings or investigate further to determine what's causing the spike.
  • If you see a spike or rise in data, sort by source type or host to understand if there is a specific cause for the increase.
  • You may then need to investigate your host or source to determine if there is an issue.
  • If you don't see spikes or a higher trend line, you do not have an issue with ingestion.


The graphic shows the Index Details for an index to illustrate a spike in activity.

Check your data quality

This topic discusses how to check the quality of your data and how to repair issues you may encounter. However, the concept of data quality depends on what factors you use to judge quality. For the purposes of this document, data quality means that the data is correctly parsed.

Your data quality can have a great impact on both your system performance and your ability to achieve accurate results from your queries. If your data quality is degraded enough, it can slow down search performance and cause inaccurate search results. Therefore, it's important to take the time to check and repair any data quality issues before they become a problem.

Generally, data quality issues fall under three main categories:

  • Line breaks. When there are problems with line breaks, the ability to parse your data into the correct separate events that it uses for searching is affected.
  • Time stamp parsing. When there are timestamp parsing issues, the ability to determine the correct time stamp to use for the event is affected.
  • Aggregation. When there are problems with aggregation, the ability to break out fields correctly is affected.

Guidelines

Finding and repairing data quality issues is unique to each environment. However, using these guidelines can help you address your data quality.

  • It's a good idea to check your most important data sources first. Often, you can have the most impact by making a few changes to a critical data source.
  • Data quality issues may generate hundreds or thousands of errors due to one root cause. Therefore, it is recommend that you sort by volume and work on repairing the source that generates the largest volume of errors first.
  • Repairing data quality issues is an iterative process. Repair your most critical datasources first, and then run queries against the source again to see what problems remain.
  • For your most critical source, you should ideally resolve all data quality issues. This helps to ensure that your searches are effective and your performance is optimal.
  • Run these checks on a regular cadence to keep your system healthy.

Example

The following example shows the process of resolving a common data quality issue. The steps to resolve your data quality issues may differ, but you can use this example as a general template for resolving data quality issues.

  1. Go to Cloud Monitoring Console > Indexing > Data Quality to see the Data Quality dashboard.


  2. View the Event Processing Issues by Source Type dashboard. In this example, you can see that the greatest volume of issues are timestamp parsing issues in the splunk_python source. Since the splunk_python source has the most errors, and most are timestamp errors, we decide to work on timestamp errors. The steps below show you how to resolve timestamp errors.


  3. The graphic shows the Cloud Monitoring Console > Indexing > Data Quality page detail. It is intended to orient the user.

  4. In this example, we are most concerned with timestamp errors in the syslog source, so we drill down into that source. Drilling down, we can see that the majority of issues are with the following source: /var/log/suricata/stats.log.


  5. The graphic shows the Cloud Monitoring Console > Indexing > Data Quality page with a detailed view of syslog. This is a troubleshooting step for repairing timestamp issues.

  6. Clicking the source allows us to drill down further, where we can see searches against this source.


  7. The graphic shows a Cloud Monitoring Console > Indexing > Data Quality detail. We can see the detailed search query and details about the search. It is a troubleshooting step in repairing timestamp issues.

  8. From here, we can look at a specific event. We can see that the issue is that Splunk was unable to parse the timestamp in the MAX_TIMESTAMP_LOOKAHEAD field.


  9. The graphic shows a Cloud Monitoring Console > Indexing > Data Quality detail. From this detail, we can see that the timestamp in the MAX_TIMESTAMP_LOOKAHEAD field needs to be repaired.

  10. To fix this, we go to Settings > Source types.


  11. In the filter, enter syslog for the source type.


  12. This graphic shows a search for syslog in the Settings > Source types page. It is intended to show a troubleshooting step in repairing a timestamp in syslog source.

  13. Select Actions > Edit. The Edit Source Type page opens.


  14. Click Timestamp > Advanced… to open the Timestamp page for editing. Ensure you are satisfied with timestamp format and the Lookahead settings. In this case, we need to edit the Lookahead settings so that Splunk can parse the timestamp correctly.


  15. The graphic shows the process of editing a timestamp in the Settings > Sourcetype screen in Splunk Cloud. It is intended to illustrate editing the timestamp.

  16. Returning to the main Edit Source Type page, go to the Advanced menu. From here you can make other changes if needed.


  17. The graphic shows the Edit Source > Advanced screen. It is intended to orient the user.

Understand your search performance

Healthy search loads are critical to the performance of your entire Splunk Cloud environment. Understanding search patterns can help you to determine if your search workload is aligned with best practices and optimized for the best performance. Often by looking more deeply into search patterns, you can see if a specific user, search, dashboard, or app is inhibiting your performance. If you encounter an issue, you can then work with users to improve performance. Search performance can be investigated by focusing on some key areas.

  • Skipped Searches
  • Search runtime

Skipped searches

If you are skipping searches, it can be indicative of problems with your search scheduling or query formation. For example, maybe you have scheduled too many searches to run at the same time, and you can alleviate the problem by staggering the scheduled searches.

You may also find that you have a search that attempts to run before the previously scheduled search has completed. For example, if you schedule Search_A to run every five minutes, but the first instance of the search takes 10 minutes to complete, then the next time the search is scheduled to run, it will be skipped because the first search has not yet completed. If this occurs, you may need to adjust the time range (set it to 10 minutes instead of 5), or you may need to optimize your search to align with search best practices to improve performance.

For more information about optimizing searches, see About search optimization.

Lastly, you may have skipped searches because your users have met the threshold for concurrency limits that you set in your Splunk System Limits. This is expected behavior, but it may also indicate that your users need help in optimizing their searches.

To check for skipped searches

  1. Go to Cloud Monitoring Console > Search > Skipped Scheduled Searches.
  2. In the Time Range field, select 24 hours to get a better picture of your searches historically.
  3. In the Count of Skipped Searches panel, sort by Reason. Frequently, there are a number of skipped searches for the same reason. Take a note of the primary reason or reasons that searches are skipped.
  4. Scroll down to see which report is generating the primary issues, and take note of the report name. If you determine that this is an expected behavior, you don't need to research any further. But, if the skipped searches are unexpected, continue to the next step.
  5. Go to Settings > Searches Reports and Alerts.
  6. If you know the App associated with the search or report, you can sort by the App. Otherwise, search by the report or search name.
  7. Once you locate the search or report, click on it to open the Edit Search dialog box.
  8. At this point, you may need to troubleshoot the formation of the search (look for wild cards, check to see if an index is specified, etc).
  9. Or, if you found that scheduling is the problem, go to Edit > Edit Schedule to review the schedule for the search.
  10. Verify that the schedule for the report or search is in line with how long the search should take to complete. For example, if the report runs every hour, but it takes 1.5 hours to run the search, the searches will be skipped.

Search runtime

When searches run for a long time, they may use too much compute and memory, causing an overall slowness of the Splunk instance, This commonly occurs when a few poorly formed searches are taking a large amount of resources. It can also occur if you have a dashboard that is being frequently used by multiple users concurrently. In each of these cases, investigating further can help you to pinpoint the searches that are long-running and determine if you can optimize them. Because each company's environment is different, it's not easy to set benchmarks for search performance. Generally, the best way to understand your search performance is to compare your historical search times with your current search times to see if there is a change. If search runtimes have slowed, review changes to your environment and new searches to determine if you need to optimize your searches or environment. For example, you may have added a poorly formed search, or you may have added a dashboard that has attracted a lot of traffic.

To find the top expensive ad-hoc searches

  1. Go to Cloud Monitoring Console > Search > Expensive Searches.
  2. Scroll down to the Top 20 Most Expensive Ad-hoc Searches.

To review searches by user

  1. Go to Cloud Monitoring Console > Search > Search Usage Statistics.
  2. Change the time frame to widen the scope. For example set it to week to date.
  3. Split the search by users so that you can see if there are a few users that are typically running longer searches.
  4. Sort by Cumulative Runtime to see which users have the most cumulative search time.
  5. Sort by Median Runtime to see which users are running the median longest searches.
  6. Click on the name of the user to drill down into more details about that user's searches.
  7. If the user running the most or longest searches is the system user, you may want to review your applications to make sure that you have optimized them, and that they are providing the expected value. You may discover that some applications are not needed or are not used.

Reviewing this data will give you a better understanding of which users run a large number of searches (or run a few long-running searches). At this point, you may want to review the searches for that user in more detail so that you can better understand if they can be optimized.


For more information about optimizing searches, see About Optimization.

To review long-running searches

  1. Go to Cloud Monitoring Console > Search > Search Usage Statistics.
  2. Expand the time range to at least 24 hours. Searches are automatically sorted by long-running searches.
  3. The Only Ad-Hoc Searches toggle should be set to no. This ensures that you will see scheduled searches, which are more likely to be long-running searches than ad-hoc searches.
  4. Scroll down to the Search Details panel where the searches are sorted by search runtime.
  5. Click the search name to view more details, and scroll to the bottom of the screen. Two events are displayed. In the second event, you can see the search query.

If you discover a long-running query that runs frequently, you may want to expand the time range to a week or longer to see how commonly this search is run. If it is running frequently, consider optimizing the search.

For more information about optimizing searches, see About Optimization

Verify your Splunk Cloud instance is ready to upgrade

When you upgrade your Splunk Cloud instance, there are dependencies that you need to address so that you can successfully upgrade. Typically, you need to verify the state of your apps and your forwarders as a part of your upgrade process. However, each upgrade overview contains specific recommendations that are relevant for your particular upgrade, so be sure to review the information for your particular upgrade. For details about current available upgrades, see the following sections:

View your upgrade readiness at-a-glance

You can view whether you are ready for an upgrade by going to Cloud Monitoring Console > Splunk Upgrade Dashboard. In the Overview section, you can see a summary view of the factors that affect whether you are ready to upgrade. These fields vary depending on your particular version.

Upgrade from 7.0.x to 7.2.x

Field Meaning
Current Installed Splunk Version Your current version of Splunk Cloud displays.
Splunk Upgrade Target Version The approved upgrade version displays.
Overall status The overall status of your Splunk Cloud instance. The field displays in green if passed and red if blocked. Blocked means that Splunk Cloud Operations cannot upgrade your Splunk Cloud instance unless you modify your apps or forwarders.
App status The overall status of your Splunk Cloud apps. The field displays Pass in green if passed and Fail in red if failed. Failed means that one or more of your apps are not ready for the upgrade. For more details about your apps, go to the Splunk App Compatibility Summary section of the dashboard.
Forwarder compatibility The overall status of your Splunk Cloud forwarders. The field displays Pass in green if passed and Provisional in yellow if an upgrade is needed. Provisional means that one or more of your forwarders are not ready for the upgrade. If the status is Provisional, go to the Forwarder Count by Status section of the dashboard.
SHA The field displays Pass in green if passed and Fail in red if failed. If the SHA field fails, this means your SAML certificate does not use the SHA-256 hash algorithm. Starting in version 7.2, your SAML certificate must use the SHA-256 hash algorithm instead of the SHA-1 hash algorithm. Upgrade your SAML certificate prior to the Splunk Cloud upgrade.
DUO The field displays Pass in green if passed and Fail in red if failed. If the DUO field is red, it means you are using a non-supported DUO authentication. The solution you choose must integrate with a SAML-based authentication solution. There are numerous SAML-based authentication solutions that Splunk Cloud supports that also enables multi-factor authentication.

Upgrade from 7.2.x to 8.0.x

Field Meaning
Current Installed Splunk Version Your current version of Splunk Cloud displays.
Splunk Upgrade Target Version The approved upgrade version displays.
Overall status The overall status of your Splunk Cloud instance. The field displays in green if passed and red if blocked. Blocked means that Splunk Cloud Operations cannot upgrade your Splunk Cloud instance unless you modify your apps or forwarders.
App status The overall status of your Splunk Cloud apps. The field displays in green if passed and red if failed. Failed means that one or more of your apps are not ready for the upgrade. For more details about your apps, go to the Splunk App Compatibility Summary section of the dashboard.
Forwarder compatibility The overall status of your Splunk Cloud forwarders. The field displays Pass in green if passed and Provisional in yellow if an upgrade is needed. Provisional means that one or more of your forwarders are not ready for the upgrade. If the status is Provisional, go to the Forwarder Count by Status section of the dashboard.

Check your app compatibility

You likely have one or more apps installed on your Splunk Cloud instance. To upgrade, you might need to upgrade your apps prior to upgrading your Splunk Cloud instance.
For a detailed view of your Splunk Cloud app compatibility, go to Cloud Monitoring Console > Splunk Upgrade Dashboard > Splunk App Compatibility Summary.

You may see one of the following statuses:

Status Meaning Recommended action
Splunk Cloud Supported, App Upgrade Required These are Splunk-supported apps, and they require an upgrade before you can upgrade your Splunk Cloud instance. Contact Splunk Support to upgrade these apps prior to upgrading your Splunk Cloud instance.
App EOL The app is at end-of-life. There are two options:
  • Authorize Splunk Support to uninstall the app - this option is recommended.
  • Inform Splunk Support that you acknowledge that this app is end-of-life, but you accept the risk of keeping it installed. Choosing this option negates the ability to open P1-P3 support cases if there are issues with the app. In addition, if issues arise post-install, you may be required to uninstall the app from Splunk Cloud if the app is found to be a root cause.
Upgrade Blocker, pending compatibility This group of apps is not yet compatible with the target release upgrade. There are two options:
  • Contact Splunk Support and state that you prefer to wait for the upgrade until the app is compatible.
  • Open a support case to request that Splunk Support remove the app until an upgraded app is available.
Not Splunk Cloud Supported, App Upgrade Available Apps in this category are not Splunk Cloud vetted. Splunk is not responsible for determining whether the app is compatible with the target upgrade version of Splunk Cloud, but there is a newer version available on Splunkbase. An upgrade version is available.
Not Splunk Cloud Supported, No App Upgrade Available Apps in this category are not Splunk Cloud vetted and do not currently have a version available on Splunkbase compatible with the target version of Splunk. It is the responsibility of customers to maintain non-supported apps. Contact Splunk Support and request that the app is uninstalled.
No App Upgrade Required These apps are ready for upgrade and no action is required. No action is required.
Enterprise Security Dependency These apps are associated with Enterprise Security and are upgraded by Splunk Support at the time that Enterprise Security is upgraded. No action is required.
Unknown These app have an unknown status. While the apps are not identified as blockers, ensure you do further research to verify these apps are compatible with the target Splunk Cloud version.



Click on any field to see detailed information on your apps.

The graphic shows how to view details about apps and forwarders in the Upgrade Readiness Dashboard. It is intended to orient the user.

For more information about working with apps in your Splunk Cloud environment, see Install apps in your Splunk Cloud deployment.
For more information about working with private apps in your Splunk Cloud environment, see Manage private apps in your Splunk Cloud deployment.

Check your forwarder compatibility

Maintaining version compatibility between your forwarders and Splunk Cloud environment ensures there is no interruption to your data securely getting to Splunk Cloud. In addition, when forwarders are version compatible with your Splunk Cloud environment, you can immediately take advantage of new capabilities. As best practice, run the most recent forwarder version, even if the forwarder is a higher version number than your Splunk Cloud environment.

Forwarder compatibility is applicable to universal and heavy forwarders that communicate directly with Splunk Cloud. If you deploy an intermediate forwarder tier, Splunk Cloud only verifies the compatibility of forwarders that are directly connected to Splunk Cloud and cannot detect the compatibility of forwarders indirectly connected to Splunk Cloud. While Splunk Cloud cannot test the compatibility of indirectly connected forwarders, it is a best practice to run the most recent forwarder version for all deployed forwarders.

For a detailed view of your Splunk Cloud forwarder compatibility, go to Cloud Monitoring Console > Splunk Upgrade Dashboard > Forwarder Count by Status.

Status Meaning Recommended action
No action needed The forwarder version is compatible. No action is required.
Provisional The forwarder version is not recommended for use with the target upgrade version of Splunk Cloud. While there are no current known issues, it is not recommended to use this version. This version may be used with your target Splunk Cloud instance, but, as best practice, it is recommended that you upgrade your forwarder. Click on the field value to drilldown for information your forwarders in that category.
Upgrade needed The forwarder version is not compatible with the target upgrade version. You have two options:
  • Upgrade the forwarder prior to upgrading your Splunk Cloud instance. Click on the field value to drilldown for information your forwarders in that category.
  • Skip the forwarder upgrade and acknowledge that this forwarder is not compatible with the target upgrade version, but you accept the risk of keeping it installed.

For more information about upgrading your forwarders, see Upgrade your Forwarders.

Monitor SVC usage with the Infrastructure-based Licensing dashboard

If your Splunk Cloud subscription plan measures the search workload consumption by Spunk Virtual Core (SVC) units, Splunk Cloud administrators can use the Infrastructure-based Licensing dashboard on the Cloud Monitoring Console to monitor usage and stay within their subscription entitlement.

For more information about the infrastructure-based subscription model and how it differs from ingest-based subscriptions, see Infrastructure pricing in the Splunk Cloud Service Description.

Understand Splunk Cloud performance considerations and SVC

Splunk Cloud provides a high performance solution for customers. Every Splunk Cloud subscription plan is provisioned with adequate compute capacity. Because search workloads can vary considerably, ingest-based subscription plans with peak daily ingest of 1000 GB (1 TB) and greater are entitled to an allocation of Splunk Virtual Cores as defined below.

An SVC is a unit of capabilities in Splunk Cloud that includes compute, memory, and I/O resources. SVCs are allocated to your ingest-based subscription plan based on your average daily ingest, up to the maximum of 1 SVC for every 10 GB of licensed peak daily ingest. Subscriptions of Premium Solutions such as Enterprise Security and IT Service Intelligence provides incremental SVC allocation of 1 SVC for every 20 GB of licensed peak daily ingest.

The ratio of allocated SVC to licensed peak daily ingest level is subject to change with the evolving infrastructure and architecture of the service. Splunk Cloud establishes SVC performance using a Splunk Search Benchmark to ensure that new ratios continue to provide the same or better levels of performance.

View the Infrastructure-based Licensing dashboard

The Infrastructure-based Licensing dashboard contains five panels visible to Splunk Cloud administrators that show SVC usage (either ingest-based or infrastructure-based subscriptions) over a specific time range.

The SVC Usage and Index Disk Usage are overview panels that display your data utilization against your Splunk subscription entitlement limits. If your utilization is consistently meets or exceeds your subscription entitlement limits, contact your Splunk Sales representative to increase the number of SVCs allocated to your stack.

Three detail panels under the overview panels show the users, indexes, and searches that are high SVC consumers:

  • Top 10 SVC Consumers
  • Top 10 SVC Consuming Indexes
  • SVC Usage by Search Type

These panels help pinpoint where you need to optimize to reduce your organization's SVC consumption. For how to take action on the insights provided by each panel, see Steps to investigate your panels.

You also can set up an alert action (for example, send an email) to be performed when a platform alert is triggered. Go to Cloud Monitoring Console > Settings > Alerts setup to configure alerts, and see Set up alert actions in the Alerting Manual for more details.

Steps to investigate your panels

Follow these steps to understand your SVC usage and take the appropriate action to reduce consumption.

To ensure accuracy, do not modify the search criteria within any of the panels on this dashboard.

Steps Detailed Image
Go to Cloud Monitoring Console > License Usage > Infrastructure-based Licensing.

A blue progress bar may appear above a panel, indicating that Splunk is still generating data. Wait for the bar to disappear before reviewing the panel.

The graphic shows the Infrastructure-based Licensing dashboard of the Cloud Monitoring Console. It is intended to orient the user.
On the SVC Usage panel, the green line indicates the license limit and the blue bars indicate the peak SVC utilization.

For best performance, utilization should be at 60-70% of the license limit. If it exceeds 70%, look at the detail panels and take action to optimize the high consumers of SVC. When your utilization is 80-90% of your license limit, there is a risk of performance impact if you don't proactively manage your consumption. You can do this by reviewing the high SVC consumers or by increasing the available SVC allocation. Contact your Splunk Sales representative to discuss allocating more SVCs to your stack.



The graphic shows the SVC Usage panel in the Cloud Monitoring Console. It is intended to show a healthy SVC utilization that is within the license limit.

The Index Disk Usage panel shows your organization's storage consumption against the license limit.

If this panel indicates your indexes are high consumers of SVC, look at the Top 10 SVC Consuming Indexes panel for specific indexes that need remediation. Look also at the SVC Usage by Search Type panel to optimize the searches handled by a particular index.

The graphic shows the Index Disk Usage panel in the Cloud Monitoring Console. It is intended to show an index disk SVC consumption that is within the license limit.
The Top 10 SVC Consumers panel lists the top SVC users and their respective SVC consumption.

With this data, you can contact high consumers of SVC and discuss ways to optimize their consumption, such as improving their search queries. The Learn more link accesses the Write better searches topic that you can share with your users.



User SVC Consumption
admin 53%
cgarcia 10%
vpatel 5.9%
wzhang 4.3%
cmon_user 0.85%
mdubois 0.0004%
The Top 10 SVC Consuming Indexes panel shows the top indexes with the highest utilization.

With this data, you can identify which indexes need optimization or other remediation. For more information on maintaining indexes, see Manage Splunk Cloud indexes.



The graphic shows the Top 10 SVC Consuming Indexes panel. The number of indexes listed varies from 1-10. The legend on the right side lists each index by name and shows its assigned color.

The SVC Usage by Search Type panel shows which searches utilize the greatest SVC as a percentage of the total consumption. Search types are grouped into the following categories: Acceleration, Ad hoc, Other, Scheduled, and Summarization. Understanding SVC usage by search type can help you understand spikes in ad hoc searches, or clustering of scheduled searches. See also Set limits for concurrent searches.


The graphic shows the SVC Usage by Search Type panel. The legend on the right side lists each search type by name and shows its assigned color.

Self-service Splunk Cloud: Enable platform alerts

The Cloud Monitoring Console for self-service Splunk Cloud provides preconfigured alerts that you can enable. If a platform alert is triggered, the Cloud Monitoring Console displays a notification on the Overview dashboard. In addition, you can set up an alert action (for example, send an email) to be performed when a platform alert is triggered. See Set up alert actions in the Alerting Manual for more details.

To enable platform alerts:

  1. Go to Cloud Monitoring Console > Settings > Alerts setup
  2. Click Enable next to the alert you wish to enable. Notifications will be displayed in the Overview dashboard.
  3. To optionally set up an alert action for the alert, click Advanced Edit.
Last modified on 06 April, 2020
PREVIOUS
Splunk Cloud data policies
  NEXT
Manage Splunk Cloud indexes

This documentation applies to the following versions of Splunk Cloud: 7.1.3, 7.1.6, 7.2.4, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 8.0.2001


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters