Using RapidDiag

The RapidDiag app is provided to assist the Splunk Administrator with collecting diagnostic information from one or more Splunk Enterprise instances simultaneously. What makes RapidDiag unique from the diag command is the ability to use distributed search to run diagnostic collections across multiple nodes, while leveraging both operating system (OS) level tools and Splunk Enterprise tools to collect troubleshooting information.

When should I use RapidDiag?

RapidDiag offers a way to collect the data from OS-level tools or other sources automatically, and collect the results in one file. It is designed to ease data collection tasks when working with Splunk Support on troubleshooting an issue.

What node do I run RapidDiag from?

The RapidDiag app requires distributed search access to other Splunk Enterprise instances. In a typical Splunk Enterprise environment, there are several roles that are configured to search other Splunk Enterprises instances:

• Monitoring Console: The monitoring console is typically configured with search access to the entire Splunk Enterprise deployment. This allows RapidDiag collections to access to the search tier, indexers or cluster peers, and supporting nodes such as the cluster manager node.
• Manager node: The cluster manager node is configured with search access to the cluster peers.
• Search Head: A search head is configured with search access to the indexers or cluster peers.

The RapidDiag app includes command line support (CLI) and help. Use splunk cmd rapidDiag -h to review the supported CLI commands. However, the CLI is for single instance use only.

There is no RapidDiag support for universal forwarders.

How do I access RapidDiag?

The RapidDiag UI is located in the Settings menu, under System > RapidDiag.

The RapidDiag app has several requirements:

• The RapidDiag app is included with Splunk Enterprise 8.1.1 and later.
• The RapidDiag app is available on Linux-based Splunk Enterprise installations only.
• A user must have the get_diag capability to access the RapidDiag UI.

Accessing the internal reference guide

The RapidDiag UI offers a reference guide in product. The Reference Guide tab provides details on folder paths used for common tools, OS tool dependancies, and Linux distribution compatibility.

Using a task template

In RapidDiag, a task template is a series of data collection tasks bundled together and named for their troubleshooting use case. The data collection tasks define OS and Splunk Enterprise tools used to collect the data. For example, the "File reading" template will generate multiple data collection tasks using the tools: iostat, ps, strace, diag, and others.

A peer node is the Splunk Enterprise instance where you want to perform a data collection task. You must select a peer node before choosing a task template. If the node where you're running RapidDiag is configured for distributed search across other Splunk Enterprise instances, you can select one or more peer nodes to run a task template on.

Monitoring a running task

The Task Manager tab in RapidDiag displays the active and historical task collection jobs. Once a collection is finished, you will see the output file path with a custom folder name used to store the data archive on the machine where the collection ran.

When a task collection is run on remote peer nodes, the data is stored on those nodes. RapidDiag does not move or copy the archive files to a central collection point. You must collect the archives from each peer node manually using the output file path reported in the completed task collection.

A troubleshooting example

Splunk Support has asked you to run the "Indexer health" template on all indexers to assist them in troubleshooting an issue.

1. Select a Splunk Enterprise node to run RapidDiag on. In this case, a search head is ideal as it has distributed search configured to search all of your indexers.
2. Log into SplunkWeb on the search head using the Splunk administrator credentials.
3. Open RapidDiag.
4. On the Task Templates page, select your indexers in the Peer Node dropdown.
5. Choose the "Indexer Health" template. Select "Next."
6. On the Review page, review the settings for the collectors.
7. Select "Start Collecting."
8. On the Task Manager page, wait for the job status to change from "Collecting" to "Success."
9. Copy the Output File path from the completed collection, and use it to copy the archive files from each indexer to a central location where you'll upload them to a support case.