Searches power dashboards and forms

Splunk searches power dashboards, forms, and the visualizations of data that they contain. This topic provides an overview of the types of searches available to you and how to include them in dashboards and panels using Simple XML.

Overview of searches in dashboards

There are several ways to access searches that drive the content of a dashboard.

Inline searches

An inline search is a search you create within a dashboard or visualization.

You can provide inline searches that are global to a dashboard or provide inline searches for each visualization in a dashboard. Searches that are global to a dashboard require post-process searches in visualizations. The post-process searches further modify the data returned from the global search.

See Inline search example.

Searches saved as reports

You can save a search as a report and access the search in a dashboard by reference to the report. See Create and edit reports in the Reporting Manual for details.

See the example, Reference a search from a report.

Generate searches with Pivot

Use Pivot to generate searches as pivots that you can export to dashboards. For more information, see the Pivot Manual. The chapter Design pivot tables with the Pivot Editor provides details on building and exporting pivots as searches.

Searches to populate form inputs

You can use searches to dynamically populate choices for form inputs such as radio buttons, drop-down lists, and check boxes.

See the example, Populate choices for form inputs.

Use tokens with searches

Searches can access tokens, a type of variable that references search fields and their values. In the search command, surround a field with $...$ characters to define a token. In the code snippet below, a token had been previously defined with $series_tok$.

index=_internal source=*metrics.log group="per_sourcetype_thruput" series=$series_tok$ | table sourcetype eps, kb, kbps

Use the token in a form to accept user input and to display labels and titles in dashboards. The Basic form example shows how to use tokens within forms. See also Token usage in dashboards.

Simple XML elements for searches in dashboards

Use the <search> element and its child elements to define searches in simple XML. The <query> element provides the actual search string. The <earliest> and <latest> elements provide the bounds of the search.

Use the search element in the following contexts:

• A search that drives the data for a visualization.

• Global search for a dashboard or form.
  Use a post-process search in visualizations to reference the global search.

• Post-process search for a visualization.
  The post-process search modifies the data returned from a global search.

• A search that provides the labels and value for inputs such as a radio input or dropdown input.

See the Search element in the Simple XML Reference for details on writing searches in simple XML code.

Search examples in simple XML

This topic provides examples of using the <search> element in the following contexts:

• Inline search.
• Reference to a search from a report.
• Populate choices for an input.
• Post-process searches that access a global search.

Inline search example

The search in this example drives the data in the visualization.

• <query>
  Provides the search string.
• <earliest> <latest>
  Define the bounds for the search.

<dashboard>
  <label>Visualization with inline search</label>
  <description></description>
  <row>
    <panel>
      <chart>
        <title>Radial gauge</title>

        <search>
          <!-- Inline search query -->
          <query>
            index=_internal source="*splunkd.log"
            ( log_level=ERROR OR log_level=WARN*
            OR log_level=FATAL OR log_level=CRITICAL )
            | stats count as log_events
             | rangemap field=log_events low=1-100 elevated=101-300 default=severe
          </query>

          <!-- search bounds -->
          <earliest>-7d@h</earliest>
          <latest>now</latest>
        </search>

        <option name="charting.chart">radialGauge</option>
        <option name="charting.chart.rangeValues">[0,300,600,900]</option>
      </chart>
    </panel>
  </row>
</dashboard>

Reference a search from a report

The searches in this example reference a report.

You cannot modify the search from the dashboard, but you can modify the time bounds and the visualization for the search results. If the search in the report changes, the visualization based on that report updates to include the changes.

You can also reference a scheduled report in a dashboard panel. When the dashboard loads, panels backed by scheduled reports load instantly with the results from the last scheduled run of the report. This practice can improve the dashboard user experience in cases where you have searches that ordinarily take a long time to run. It can also reduce the search processing load on your system if you have dashboards that are loaded frequently by large numbers of users. See "Add panels to dashboards" in this manual.

The report in this example uses a bar chart for the visualization and displays results for the last seven days. The panel on the left displays the search from the report. The panel on the right uses the same search from the report, but modifies the time bounds and visualization.

<search> element code:

• <search ref ="[name]">
  References the report.
• <earliest> <latest>
  Modify the time bounds.

<dashboard>
  <label>Search from report</label>
  <row>
    <panel>
      <title>Original report</title>
      <chart>
        <title>Source types time chart</title>
        
        <search ref="Source types time chart" />
        
      </chart>
    </panel>
    <panel>
      <title>Modified time bounds and visualization</title>
      <chart>
        <title>Source types time chart</title>
        
        <search ref="Source types time chart">

          <!-- Modify time bounds -->
          <earliest>-30d@d</earliest>
          <latest>now</latest>

        </search>

        <!-- Change visualization -->
        <option name="charting.chart">column</option>

      </chart>
    </panel>
  </row>
</dashboard>

Populate choices for form inputs

Use the search element to dynamically populate the choices for the following form inputs:

• Check boxes
• Drop-down list
• Multiselect input
• Radio buttons

Caution: Do not use a real-time search for a populating search. The input choices do not update correctly when using a real-time search.

The search in this example compares static and dynamic definition for choices. The drop-down list uses a populating search to define the choices.

• Populating <search>
  Returns fields to use for the label and value of the choices.

• <fieldForLabel> <fieldForValue>
  Child elements to the <input> element. These elements specify the fields to use to populate choices for the dropdown.

<form>
  <label>Populate an input with a search</label>
  <description>Events Filtered by User and Sourcetype</description>
  <!-- Do not need a Search Button. Inputs search when changed -->
  
  <fieldset autoRun="true" submitButton="false">
    
    <!-- Static definition of choices -->
    <input type="radio" token="username_tok" searchWhenChanged="true">
      <label>Select a User:</label>
      
      <!-- Define the default value -->
      <default>All</default>

      <!-- Hard-code the choices -->
      <choice value="*">All</choice>
      <choice value="-">-</choice>
      <choice value="admin">Admin</choice>
      <choice value="nobody">Nobody</choice>
      <choice value="splunk-system-user">Splunk System User</choice>
    </input>
    
    <!-- Dynamic definition of choices -->
    <input type="dropdown" token="sourcetype_tok" searchWhenChanged="true">
      <label>Select a Sourcetype:</label>
      <prefix>sourcetype="</prefix>
      <suffix>"</suffix>
      
      <!-- Define the default value -->
      <default>splunkd</default>
      
      <!-- Hard-code the choice for "All" -->
      <choice value="*">All</choice>
      
      <!-- Define the other choices with a populating search -->
      <search>
        <query>
          index=_internal | stats count by sourcetype
        </query>
      </search>
      <fieldForLabel>sourcetype</fieldForLabel>
      <fieldForValue>sourcetype</fieldForValue>
    </input>
    
  </fieldset>
  <row>
    <panel>
      <!-- Use tokens from the <input> elements in the panel title -->
      <title>
        Input selections: (radio) "$username_tok$", (dropdown) $sourcetype_tok$
      </title>
      
      <chart>
        
        <!-- search for the visualization, references the input tokens-->
        <search>
          <query>
            index=_internal user=$username_tok$ $sourcetype_tok$ | timechart count
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
        </search>
       </chart>
      
    </panel>
  </row>
</form>

Post-process searches

Sometimes you end up with a dashboard running searches that are similar. You can save search resources by creating a base search for the dashboard. Panels in the dashboard use a post-process search to further modify the results of a base search. The base search can be a global search for the dashboard or any other search within the dashboard.

Typically, the global search is a transforming search. A transforming search uses transforming commands to transform event data returned by a search into statistical data tables. See transforming commands and searches in the Search Manual.

Be aware of the limitations for post-process searches that arise from the following causes:

• Base searches returning more than 500,000 events.
• Splunk Web time-out from search operations that exceed 30 seconds to complete.

See Post process limitations for details on these limitations and other cautions about using post-process searches. The topic Post-process examples provides guidance on constructing post-process searches.

Post-process limitations

Post-process searches have limitations. If you do not use a transforming base search, these limitations can cause data truncation or performance issues.

Search result count limit
In non-transforming base searches, the Splunk platform retains only the first 500,000 events returned. A post-process search does not process events in excess of this 500,000 event limit, silently ignoring them. This results in incomplete data for the post-process search. A transforming base search helps avoid this limitation.

Note: This search result retention limit matches the max_count setting in limits.conf. This setting defaults to 500,000.

Timeout
If the post-processing operation takes too long, it can exceed Splunk Web client’s non-configurable timeout value of 30 seconds. This can result in a timeout due to an unresponsive splunkd daemon/service. This scenario typically happens when you use a non-transforming search as the base search.

Avoid base searches that return raw events

Avoid using a base search without transforming commands. If a base search returns raw events in excess of event limitation, incomplete data might be passed to post-process searches. Use transforming commands in the base search to avoid the event limitation. See About transforming commands and searches in the Search Manual.

Avoid post-process searches that reference fields not named in the base search

It might seem logical to reference a field only in the post-process searches, but it is better to isolate the data for the field in the base search. Otherwise, the field that is referenced only in the post-process search becomes null in all rows, thus returning zero results.

Avoid this problem by using transforming commands in the base search.

Avoid returning large numbers of rows in the base search

Passing a large number of search results to a post-process search can cause problems.

Server time out

If the post-processing operation takes too long, it can result in performance problems and possibly a timeout. In this scenario, consider the following:

• The number of results and fields returned from the base search.
• The complexity of the post-process operations on these results.

Incomplete data

If the base search is a non-transforming search that returns in excess of the event limitation, an incomplete data set is passed to downstream panels (as described above). To avoid event limitation, use transforming commands in the base search to structure results.

Post-process examples

Post-process works best when you reformat results from a base search that uses transforming commands.

This lets you create tables and charts according to specific criteria. For example, you can create different visualizations and reports from the same data set. You can also do further aggregation on the original report.

Basic post-process example

This example uses transforming commands for the base search, post-processing the results differently:

Base search
index=_internal source=*splunkd.log | stats count by component, log_level

Post process 1 (event count by log_level)
| stats sum(count) AS count by log_level

Post process 2 (error count by component)
| search log_level=error | stats sum(count) AS count by component

<dashboard>
  <label>Dashboard with post-process search</label>

  <!-- Example uses stats transforming command -->
  <!-- This limits events passed to post-process search -->
  <search id="baseSearch">
    <query>
      index=_internal source=*splunkd.log | stats count by component, log_level
    </query>
  </search>

 <row>
    <panel>
      <chart>
        <title>Event count by log level</title>

        <!-- post-process search -->
        <search base="baseSearch">
          <query>
            stats sum(count) AS count by log_level
          </query>
        </search>

      </chart>
    </panel>
    <panel>
      <chart>
        <title>Error count by component</title>

        <!-- post-process search -->
        <search base="baseSearch">
          <query>
            search log_level=error | stats sum(count) AS count by component
          </query>
        </search>

        <option name="charting.chart">bar</option>
      </chart>
    </panel>
  </row>
</dashboard>

Chained post-process example

Chain two or more post-process searches together. The following example shows how to link one post-process search to another one.

<search id="baseSearch">
   <query>index=_internal</query>
   <earliest>-60m@m</earliest>
   <latest>now</latest>
</search>

<search base="baseSearch" id="post_process_1">
   <query>sourcetype=splunkd</query>
</search>

<search base="post_process_1" id="post_process_2">
   <query>stats count</query>
</search>

Complex post-process example

For more complex base searches that include statistical aggregations such as percentiles, standard deviations, and even averages, it is better to use summary indexing commands in the base search. This facilitates building the post-process searches. Some examples of summary indexing search commands include:

• sistats
• sitimechart
• sitop
• sichart
• sirare

The summary index equivalents provide more flexibility for post-process searches. See Use summary indexing for increased reporting efficiency and About transforming commands and searches.

Base search
index=_internal | eval event_size=len(_raw)
| sistats count min(event_size) avg(event_size) max(event_size)
by source sourcetype

Post process 1
| stats count

Post process 2
| stats avg(event_size) by sourcetype

Post process 3
| stats count by sourcetype

The base search reports event size (min, avg, max) by source and sourcetype for the _internal index. Use the sistats count with the various group-by clauses. You lose the benefits of map-reduce in distributed search if you do not include these.

<dashboard>
  <label>Dashboard with post process using summary indexing</label>


  <!-- Use summary indexing transforming command -->
  <search id="baseSearch">
    <query>
      index=_internal | eval event_size=len(_raw)
      | sistats count min(event_size) avg(event_size) max(event_size)
        by source sourcetype
    </query>
  </search>

  <row>
    <panel>
      <single>
        <title>Total event count</title>

        <!-- post-process search -->
        <search base="baseSearch">
          <query>stats count</query>
        </search>

        <option name="beforeLabel">Total events: </option>
      </single>
    </panel>
    <panel>
      <chart>
        <title>Average event size by source type</title>

        <!-- post-process search -->
        <search base="baseSearch">
          <query>stats avg(event_size) by sourcetype</query>
        </search>

        <option name="charting.axisY.scale">log</option>
      </chart>
    </panel>
    <panel>
      <chart>
        <title>Event count by source type</title>

        <!-- post-process search -->
        <search base="baseSearch">
          <query>stats count by sourcetype</query>
        </search>

        <option name="charting.axisY.scale">log</option>
      </chart>
    </panel>
  </row>
</dashboard>

Form with post-process search for inputs

You can use a post-process search to dynamically populate inputs to a form. The following example shows a form with two inputs. The drop-down list, which selects an index to search, defines the choices statically. The drop-down list to select a source type statically defines the default choice but uses a post-process search to dynamically define the other choices.

Base search for populating the source type dropdown
index=_internal | stats count by sourcetype

Post process for dropdown input
| search sourcetype=splunkd*

<form>
  <label>Post Process in Form Inputs</label>
  
  <!-- Global search for post process by dropdown input -->
  <!-- Search uses stats command to limit results -->
  <search id="searchInput">
    <query>index=_internal | stats count by sourcetype</query>
    <earliest>-60min</earliest>
    <latest>now</latest>
  </search>
  
  <fieldset submitButton="false">
    
    <!-- Select an index from two static choices -->
    <input type="dropdown" token="index_tok" searchWhenChanged="true">
      <label>Select an index to search</label>
      <choice value="_internal">Internal</choice>
      <choice value="*">All public indexes</choice>
      <default>_internal</default>
    </input>
    
    <!-- Dynamically populate choices -->    
    <input type="dropdown" token="sourcetype_tok" searchWhenChanged="true">
      <label>Select a source type</label>
      
      <!-- default choice is all sourcetypes -->
      <choice value="*">All sourcetypes</choice>
      <default>*</default>
      
      <!-- Post-process search to dynamically populate choices -->
      <search base="searchInput">
        <query>search sourcetype=splunkd*</query>
      </search>
      <fieldForLabel>sourcetype</fieldForLabel>
      <fieldForValue>sourcetype</fieldForValue>
      
    </input>
    <input type="time" token="time_tok" searchWhenChanged="true">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <title>Chart</title>
        <search>
          <query>
            index=$index_tok$ sourcetype=$sourcetype_tok$ | timechart count
          </query>
          <earliest>$time_tok.earliest$</earliest>
          <latest>$time_tok.latest$</latest>
        </search>
      </chart>
    </panel>
  </row>
</form>

Troubleshoot referenced real-time searches in search head clusters

In a search head clustering (SHC) deployment, if you are referencing a real-time saved search in a dashboard on a search head, the real-time search might not continue to stream data after initial results are returned.

There are two workarounds for this issue.

Option	Example dashboard source code	Performance considerations
Use an inline real-time search in the dashboard panel instead.	<search> <query>index=_internal | stats count </query> <earliest>rt-5m</earliest> <latest>rtnow</latest> </search>	This type of search runs only when users view the dashboard. However, a new real-time search spawns for each user that accesses the dashboard from the search head or another member.
Create a scheduled saved search. Use the loadjob command in an inline panel search to update the dashboard with the saved search results.	<search> <query> | loadjob savedsearch="admin:search:SavedSearch" </query> </search>	Only one instance of the saved search runs at the scheduled time regardless of the number of users accessing the dashboard.

Additional search resources

If you are new to the Splunk platform and the search processing language (SPL), start with the Search Tutorial. This tutorial introduces you to the Search and Reporting application. The tutorial guides you through adding data to your Splunk deployment, searching your data, and building simple reports and dashboards.

The Search Manual includes detailed information about creating and optimizing searches, retrieving events, specifying time ranges, and using subsearches.

The Search Reference is a reference guide for the Search Processing Language (SPL). The Search Reference contains a catalog of the search commands with syntax, descriptions, and examples.

---