Splunk Cloud Platform

Splunk Cloud Platform Admin Manual

Use the Alerts dashboard

CMC provides preconfigured platform alerts for missing forwarders and skipped searches that you can enable. You can also create custom platform alerts using the global Searches, Reports, and Alerts page accessible through the CMC Alerts functionality.

When a CMC platform alert is triggered, a message alert displays on registered mobile devices that are equipped with a Splunk Mobile app for Splunk Cloud Platform administrators. The alert does not display in Messages in the top Splunk Cloud bar in Splunk Web.

Splunk Cloud Platform administrators can also review alerts on the Triggered Alerts page of the CMC app and the Alerts count column on the Searches, Reports, and Alerts page.

You must be on at least app version 2.1.1 to use the CMC platform alerts functionality. To check the app version, select Support & Services > About. The CURRENT APPLICATION area at the bottom of the About page shows the app's version and build numbers.

Review triggered alerts

To view triggered alerts:

  1. In the CMC navigation bar, select Alerts > Triggered Alerts.
  2. The page displays the name of any triggered alert and a timestamp of when it was triggered.

When a preconfigured alert is triggered, CMC displays an alert with a 3 severity level on the Triggered Alerts page, which indicates medium severity.

Starting with CMC 2.6.0, preconfigured alerts use the prefix CMC. Alerts with the prefix SIM are retained for backwards compatibility.

The table describes the situations that trigger a preconfigured alert and the CMC dashboards to review to take further action.

Preconfigured alert Description Dashboards
CMC Alert - 503 errors Triggers when the server returns a 503 error from attempting to process an HTTP Event Collector (HEC) request. See Review the HTTP Event Collector dashboard to view your HEC functionality status.
CMC Alert - Bucket size and range Triggers when an index meets any of the following critical thresholds:
  • 10% of that index's buckets are quarantined
  • More than 50% of buckets on an index are less than half the max size of that bucket
  • Less than 50% of buckets have reached their full size
See Use the Health dashboard to learn more about bucket size and range health.
CMC Alert - Cache activity transfer Triggers when bucket download size is higher than 10% of total disk space on all indexers. This is the critical threshold for cache activity transfer. See Use the Health dashboard to learn more about maintaining bucket size download rate.
CMC Alert - Heavy forwarder software version Triggers when less than 15 days are remaining before end of support for the Heavy forwarder. See Use the Health dashboard to learn more about maintaining your Heavy forwarder software version.
CMC Alert - High memory searches Triggers when a search size uses more than 10% of your Splunk platform instance memory. This is the critical threshold for search memory usage. See Use the Health dashboard to learn more about optimizing searches.
CMC Alert - Indexers blocked queues Triggers when 50% or more of stack indexers are blocked from processing. See Review the Indexing Performance dashboard to investigate blocked indexer queues.
CMC-Alert - Ingest volume exceeds 80% of entitlement value Triggers when your ingest volume exceeds 80%. See Monitor current usage of your ingestion-based subscription to learn more about monitoring your ingest volume.
CMC Alert - New Data in Index Specified as "lastchanceindex" Runs at 12 minutes past midnight every day and is triggered if there is new data in the index specified as the lastchanceindex in the last 24 hours. See the following:
CMC Alert - S3 scanned volume exceeds 80% of the entitlement value Triggers when your Federated Search for Amazon S3 data scan entitlement usage exceeds 80% See Documentation:SplunkCloud:Admin:MonitoringLicenseUsage to learn more about monitoring your federated search for Amazon S3 data scan entitlement.
CMC Alert - Skipped search percentage Triggers when a search head has a skip search ratio higher than 25%. See Use the Health dashboard to learn more about lowering your skip search ratio.
CMC Alert - Storage Capacity Exceeds 80% Runs at 4:16 AM every day and is triggered if the searchable storage usage percent value for your deployment exceeds 80%. See the table in Review the Searchable Storage (DDAS) dashboard, especially the Searchable Storage Usage Percent panel description.
CMC Alert - SVC Utilization Exceeds 80% for 3 Hours Runs every hour at 12 minutes past the hour and is triggered if the SVC utilization value for your deployment exceeds 80% over a 3-hour timespan. See the table in Review the Workload dashboard, especially the SVC Usage panel description.
CMC Alert - Universal forwarder software version Triggers when less than 15 days are remaining before end of support for the Universal forwarder forwarder. See Use the Health dashboard to learn more about maintaining your Universal forwarder software version.
SIM Alerts - Missing Forwarders Runs every 15 minutes and is triggered if there are any forwarders with a status of Missing. See the Forwarders: Deployment dashboard, especially the Missing Forwarder Alerts and Status and Configuration - As of <current_timestamp> panels.
SIM Alerts - Skipped Searches Runs every 60 minutes and is triggered if the number of skipped searches exceeds 20%. See the Skipped Scheduled Searches dashboard.

Review preconfigured alerts

In the CMC navigation bar, select Alerts > Configured Alerts. The table displays the preconfigured CMC alerts and any custom alerts that you or another Splunk Cloud Platform administrator configured for your organization's deployment. Last Updated shows when an alert was edited.

Select the Enabled toggle to enable or disable an alert.

Select the Mobile Alert toggle to enable or disable an alert on mobile devices. Enabling an alert automatically enables it for display for Splunk Cloud Platform administrators on Splunk Web and registered mobile devices equipped with a Splunk Mobile app. For more information on downloading and registering a Splunk Mobile app, see the following:

Select Edit to access the Searches, Reports, and Alerts page. You can view detailed information about an alert and perform specific actions, such as reviewing the alert definition and running the alert.

Do not edit the search field for preconfigured alerts.

Manage CMC Alerts on the Searches, Reports, and Alerts page

To manage CMC platform alerts on the Searches, Reports, and Alerts page, follow these steps:

  1. Access this page through one of the following methods:
    • Select the Edit link adjacent to an alert on the Alerts > Configured Alerts page in the CMC app.
    • In the Splunk Cloud bar at the top of the page, select Settings. In the KNOWLEDGE section, select Searches, reports, and alerts.
  2. Set Type to Alerts.
  3. Set App to Cloud Monitoring Console (splunk_instance_monitoring).
  4. Set Owner to All or Nobody. The CMC and SIM alerts for CMC appear.
  5. In the Actions column, select Edit > Enable.

Create custom alerts

You can also create custom platform alerts using the Searches, Reports, and Alerts page. You can access this page through one of the two methods noted in step one of Manage CMC Alerts on the Searches, reports, and alerts page. Select the New Alert button to define an alert and the corresponding action to be performed when the alert is triggered. For example, you can send an email to the email account in a Splunk Cloud Platform administrator's profile, or an alert to their registered mobile device equipped with a Splunk Mobile app.

For more information, see the following:

  • Set up alert actions in the Alerting Manual
  • The global Alert Actions page. To access this page, in the Splunk Cloud bar at the top of the page, select Settings. In the KNOWLEDGE section, select Alert actions.
Last modified on 16 July, 2024
Use the Maintenance (preview) dashboard   Use the Indexing dashboards

This documentation applies to the following versions of Splunk Cloud Platform: 8.2.2112, 8.2.2201, 8.2.2202, 8.2.2203, 9.0.2205, 9.0.2208, 9.0.2209, 9.0.2303, 9.0.2305, 9.1.2308, 9.1.2312, 9.2.2403, 9.2.2406 (latest FedRAMP release)


Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters