Developing Dashboards, Views, and Apps for Splunk Web

Use one search for a whole dashboard

Sometimes you end up with a dashboard running various searches that are similar. You can save search resources by creating a dashboard in advanced XML that feeds all downstream panels with one single search. This topic shows how to use one base search for a dashboard, and use the HiddenPostProcess module to process the search differently for each panel.

HiddenPostProcess module

You can pass events or results to a post process search. However, Splunk recommends that you use a reporting search that passes results. The Post process search example provides an example of how to construct a data cube with your search, and pass results for post processing.

Post process limitations

Be aware of the following limitations when using post process.

A post process search has an unconfigurable limit of 10,000 raw events that can be passed to it. Events in excess of this 10,000 event limit are not processed and silently ignored, resulting in incomplete data reported for the post process search.

Passing a large number of search results from a data cube to a post process search can cause a server time out. In this scenario, consider the following:

• The number of results and fields returned from the base search.
• The complexity of the post process operations on these results.

If the post-processing operation takes too long, it can exceed Splunk Web client’s non-configurable timeout value of 30 seconds. This can result in a timeout due to an unresponsive splunkd daemon/service.

About the base search

Post process is best to use for stats instead of collecting raw events. It allows you to reformat reporting results from the search. This means you can create tables and charts according to specific criteria. For example, you can create different visualizations and reports from the same data set. You can also do further aggregation on the original report.

Only use post process on a base search that is a reporting search. You can mangle your results if you do not construct your base search correctly. Splunk recommends that you use summary indexing commands in the base search. This facilitates building the post process searches. Some examples of summary indexing search commands are:

• sistats
• sitimechart
• sitop
• sichart
• sirare

Note: You can also use primary reporting commands, such as stat, timechart, top, chart, and rare. But the summary index equivalents provide more flexibility for post process searches.

For more information see Use summary indexing for increased reporting efficiency and Use reporting commands.

Display results of a post process search

Modules that support the display of results from a post process search:

• SingleValue
• SimpleResultsTable
• EventsViewer
• JSChart
• FlashChart

Modules not supported for display of post process results:

• MultiFieldViewer
• ResultsHeader
• SimpleResultsHeader
• FlashTimeline
• SuggestedFieldViewer.

Post process search example

Construct your base search

For the base search, Splunk recommends that you build a data cube using a reporting command that produces results that can be easily repurposed by post process searches.

The following search reports event size (min, avg, max) by source and sourcetype for the _internal index:

Base search

index=_internal | eval event_size=len(_raw) | sistats count min(event_size) avg(event_size) max(event_size) by source sourcetype

The sistats count with the various group-by clauses is important. Without these specified in the search you lose the benefits of map-reduce in distributed search.

Base searches that return raw events

When you build your base search, it is tempting to build a simple search that feeds raw events to the post process searches in downstream panels. However, this does not work. The sheer number of raw events in this type of search can easily surpass the 10,000 event limit that can be passed to a post process search.

Caution: Base searches that return in excess of 10,000 raw event can result in incomplete results passed to the post process search. See Post process limitations.

Base searches that return an overwhelming number of results

It is also tempting to build a search that returns an overwhelming number of results and fields. This can cause the server to time out during the post process search. Be careful when constructing the base search and the complexity of operations during post process.

Add chrome

First, add the chrome and nav for your view:

<view template="dashboard.html">
  <label>Post process examples</label>
  <module name="AccountBar" layoutPanel="appHeader"/>
  <module name="AppBar" layoutPanel="navigationHeader"/>
  
  <module name="Message" layoutPanel="messaging">
    <param name="filter">*</param>
    <param name="clearOnJobDispatch">False</param>
    <param name="maxSize">1</param>
  </module>
  
  <module name="TitleBar" layoutPanel="viewHeader">
    <param name="actionsMenuFilter">dashboard</param>
  </module>
  . . .
</view>

Add the base search

Use the HiddenSearch or HiddenSavedSearch modules to specify the base search.

. . .
<module name="HiddenSearch" layoutPanel="panel_row2_col1" autoRun="True">
  <param name="search">
   index=_internal | eval event_size=len(_raw) | sistats count min(event_size) avg(event_size) max(event_size) by source sourcetype
  </param>
   . . .
   <!-- Add post process modules -->
  . . .
</module>

Post process a search

Use the HiddenPostProcess module to process the results from your base search and feed into a results module. For example, this panel displays search results in a SingleValue module:

<module name="HiddenPostProcess" layoutPanel="panel_row1_col1"
      group="Post process as single value">
  
  <param name="search">
    | stats count
  </param>

  <module name="SingleValue">
    <param name="field">count</param>
    <param name="afterLabel"> events</param>
    <param name="classField">range</param>
  </module>
  
</module>

This panel displays maximum event size by source in a bar chart:

<module name="HiddenPostProcess"
        layoutPanel="panel_row1_col2" group="Post process as bar chart">
  
    <param name="search">
      | stats avg(event_size) by sourcetype
    </param>
  
    <module name="HiddenChartFormatter">
      <param name="chart">bar</param>
      <param name="primaryAxisTitle.text">Source type</param>
      <param name="secondaryAxisTitle.text">Average event size</param>
      <param name="legend.placement">none</param>
      
      <module name="JSChart">
        <param name="width">100%</param>
        <param name="height">200px</param>
      </module>
        
    </module>        
</module>

This panel displays event count per sourcetype in a pie chart:

<module name="HiddenPostProcess"
        layoutPanel="panel_row1_col3" group="Post process as pie chart">

    <param name="search">
      | stats count by sourcetype
    </param>
  
    <module name="HiddenChartFormatter">
      <param name="chart">pie</param>
      <param name="chartTitle">Event count by sourcetype</param>
      
      <module name="JSChart">
        <param name="width">100%</param>
        <param name="height">200px</param>
      </module>
        
    </module>   
</module>

Example dashboard with post process searches

The following dashboard shows the results of the post process searches listed above.