Use the Forwarder dashboards

The dashboards accessed from the Cloud Monitoring Console > Forwarder tab provide information to Splunk Cloud Platform administrators about forwarder connections and status. This information helps you ensure your forwarders are correctly transmitting data to the indexers.

For data to appear on the forwarder dashboards, you must first configure and enable the Forwarder Monitoring Setup page.

A blue progress bar might appear above a panel, indicating that the Splunk platform is still generating data. Wait for the bar to disappear before reviewing the panel.

Do not modify any Cloud Monitoring Console (CMC) dashboard. Changing any of the search criteria, formatting, or layouts may cause inaccurate results and also override the automatic update process.

Manage the forwarder monitoring setup

The CMC Forwarder Monitoring Setup page helps Splunk Cloud Platform administrators manage your forwarder monitoring configuration. This includes periodically removing decommissioned forwarders to improve system performance.

Because they are configuration pages, the Forwarder Monitoring Setup pages for Splunk Cloud Platform CMC and Splunk Enterprise Monitoring Console are similar. For more information on understanding and using this configuration page, see About time settings and Rebuild the forwarder asset table in the Monitoring Splunk Enterprise manual.

A difference between Splunk Cloud Platform CMC and the Splunk Enterprise Monitoring Console is the lookup file name. For CMC, enabling forwarder monitoring runs a scheduled search that populates the sim_forwarder_assets.csv.gz lookup file.

Review the Forwarder Monitoring Setup page

To investigate this page, go to Cloud Monitoring Console > Forwarders > Forwarder Monitoring Setup.

After upgrading to CMC version 3.22.0, select '''Rebuild forwarder assets''' to ensure the data displays accurately.

The top section of the page is where you set whether forwarder monitoring is enabled and the data collection time interval, or disable it. Complete the following steps to enable forwarder monitoring.

1. Select Enable.
2. Choose a time option in Data Collection Interval.
3. Select Save.
4. Choose an option in the the Build Forwarder Assets Now dialog box that appears.
   1. Select Continue to start the forwarder assets rebuild process. This process lets you immediately rebuild the forwarders assets table, which removes decommissioned forwarders from the deployment and improves performance. Messages appear that indicate the state of this process and when it completes.
   2. Select Later if you want the forwarder assets table automatically rebuilt during the next daily update process.

Select Reset to reset the forwarder monitoring setup back to the previous configuration.

Be sure to select Save after making any configuration changes.

The bottom section of the page lets you immediately rebuild the forwarder assets table to remove any decommissioned forwarders. Complete the following steps.

1. Select Rebuild forwarder assets...
2. Choose an option in Time Range.
3. Select Start Rebuild.

Depending on the number of forwarders in your deployment, rebuilding the forwarder assets table can affect indexer performance and take a significant amount of time to complete.

Monitor forwarder instances

The CMC Forwarders: Instance dashboard provides information to Splunk Cloud Platform administrators about the status and health of the forwarders in your deployment.

Review the Forwarders: Instance dashboard

This dashboard contains two panels with tabular and graphical data for a specified forwarder instance. Set a time range to filter the results.

To investigate your panels, go to Cloud Monitoring Console > Forwarders > Forwarders: Instance. Use the following table to understand the dashboard interface.

Panel or Filter	Description
Instance and Time Range	Specify a forwarder instance and a time range. These settings apply to both panels in the dashboard. When you view this dashboard, the Instance field is automatically populated with the first menu value. Be sure to change this default value to the forwarder instance you are investigating.
Status and Configuration	Lists the following information for the specified forwarder: GUID Forwarder type IP address Splunk version OS and architecture Receiver and connection counts Average kilobytes per second and events per second
Outgoing Data Rate	Shows a graph that compares events per second and KB per second processed by the instance over the selected time range. Select an Aggregation value of either Maximum or Average.

Interpret forwarder instance results

When interpreting your forwarder instance results, note the following:

• Check that your forwarder's version is up-to-date.
• Use the IP address information to identify any faulty receivers in your local network.
• Compare the receiver count against the number of deployed indexers. A significant difference in these numbers indicates that there is likely a misconfiguration in the system.
• Review the graph in the Outgoing Data Rate panel and ensure that the forwarder is emitting data within its normal expected range. In particular, check the rates for average KB per second and events per second against their historical average rates. A rate that is significantly different from this historical rate, such as being very high or very low, could indicate an issue on the forwarding host.

Monitor forwarder deployments

The CMC Forwarders: Deployment dashboard provides comprehensive information to Splunk Cloud Platform administrators about the status and health of the forwarders in your deployment. You can also set alerts that trigger if a forwarder is missing from the deployment.

Review the Forwarders: Deployment dashboard

This dashboard shows both current status and historical information for your forwarder deployments, with various filters so you can further refine the results. Use the top panel to enable or disable missing forwarder alerts.

This dashboard contains one panel with a variable in the title: Forwarders by <variable>.

To investigate your panels, go to Cloud Monitoring Console > Forwarders > Forwarders: Deployment. Use the following table to understand the dashboard interface.

Panel or Filter	Description
Missing Forwarder Alerts	Select enable to open this panel. Specify a Filter by Last: option to view all missing forwarder alerts reported in that time range. Select the Scheduled Search: SIM Alert - Missing Forwarders link to access the Searches, reports, and alerts page. You can do the following for this alert: Confirm that the alert is successfully running every 15 minutes. Run the alert query on an ad hoc basis. View recently run jobs. You can also manage this alert with the CMC Alerts panel. For general information about managing alerts, see the Splunk Cloud Platform Alerting Manual.
Forwarders by <variable>	The <variable> in the panel title and the data in the pie chart graph dynamically change, based on the selected Split by option. The panel title is one of the following: Forwarders by Status Forwarders by Forwarder Type Forwarders by Splunk Version Forwarders by OS Forwarders by Architecture Total: <number> forwarders indicates the total number of forwarders in the deployment.
Status and Configuration - As of <current_timestamp>	Set criteria to filter the returned results: The Instance filter accepts an asterisk (*) wildcard. Specify a Status of All, Active, or Missing. Select the Show instances forwarding internal logs checkbox to further refine the results. Total: <number> on the left side of the table indicates the number of returned instances that meet the filter criteria. The table lists the following information: Instance Type Version OS Architecture Status Last Connected to Indexers Total KB Average KB/s Over Time Average KB/s Average Events/s
Historical Data	This area includes the Total Count of Forwarders and Forwarder Connection Count panels. The specified Time Range option set here affects both panels. Specify an Overlay option to view a bar graph of the average KB per second or average events per second over time.

Interpret forwarder deployment results

Use this dashboard to identify misconfigurations or unhealthy behavior of the forwarders, such as outliers in the forwarder deployment. Misconfigurations means forwarders are sending too much or too little data. You also want to investigate any sudden spike of missing forwarders, as this could indicate a systemic failure.

Check forwarder versions

The CMC Forwarder Versions dashboard shows the current installed version of Splunk Cloud Platform to Splunk Cloud Platform administrators and also indicates if your Splunk forwarders are outdated. Use this dashboard to determine which forwarders in your deployment are degrading its performance or have known compatibility issues with the deployed Splunk Cloud Platform version.

Review the Forwarder Versions dashboard

This dashboard provides four panels of information about your deployment and forwarders.

To investigate your panels, go to Cloud Monitoring Console > Forwarders > Forwarder Versions. Use the following table to understand the dashboard interface.

Panel or Filter	Description
Version Summary	Bar chart that shows forwarder version over forwarder count. The bars are color-coded to indicate if the forwarders are out-of-date (red) or up-to-date (green).
Current Splunk Cloud Platform Version	Shows the version number of your current Splunk Cloud Platform deployment. This version number also appears in the Support & Services > About window.
Upgrade Recommendations	Shows upgrade recommendations based on comparing the forwarder version and the Splunk Cloud Platform version. Lists the forwarder name, version, type, and recommendation.
Flagged Forwarders (Based on Version)	Shows all forwarders that have been identified as broken or not operating as expected. Lists the forwarder name and version.

Interpret forwarder version results

Use the CMC Forwarder Versions dashboard to identify which forwarders you must update as soon as possible. For more information, see Troubleshoot forwarder/receiver connection in the Splunk Cloud Platform Forwarding Data manual.