Splunk® Enterprise

Developing Views and Apps for Splunk Web

Download manual as PDF

Splunk Enterprise version 5.0 reached its End of Life on December 1, 2017. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Use one search for a whole dashboard

Sometimes you end up with a dashboard running various searches that are similar. You can save search resources by creating a dashboard that feeds all downstream panels with one single search. This topic shows how to use one base search for a dashboard, and use the HiddenPostProcess module to process the search differently for each panel.

HiddenPostProcess module

Use the HiddenPostProcess module to pass events or results from a base search to a post process search. However, Splunk recommends that you use a reporting command in your base search that orders the results of the search into a data cube. For example, the following base search returns a large number of events, which can be problematic for post processing, as described in Post process limitations:

index=_internal source=*splunkd.log

Using a reporting command to aggregate many events into a smaller number of results provides a stats-driven data cube which is more amenable to post processing:

index=_internal source=*splunkd.log | stats count by component, log_level

Read more about reporting commands in the Search Manual. The Post process search example provides an example of how to construct a data cube with your search, and pass results for post processing.

Post process limitations

Be aware of the following limitations when using post process.

  • If the base search is a non-transforming search, Splunk retains only the first 500,000 events returned. In this case, events in excess of this 500,000 limit are not processed by the post process search, resulting in incomplete data. Splunk recommends that you use reporting commands in the base search to avoid this problem.
  • If the post-processing operation takes too long, it can exceed Splunk Web client’s non-configurable timeout value of 30 seconds. This can result in a timeout due to an unresponsive splunkd daemon/service. This scenario typically happens when you use a non-transforming search as the base search. Splunk recommends that you use reporting commands in the base search to avoid this problem.

Avoid base searches that return raw events

It might seem logical to have a non-transforming base search that returns raw events, and then use transforming commands in the post process search. However, in this scenario the base search could return in excess of the 500,000 event limitation. Thus, it passes an incomplete data set to the post process searches, producing erroneous results in your dashboard.

It is better to use reporting commands in the base search to avoid the 500,000 event limitation.

Avoid post process searches that reference fields not named in the base search

It might seem logical to reference a field only in the post process searches, but it is better to isolate the data for the field in the base search. Otherwise, the field that is referenced only in the post process search becomes null in all rows, thus returning zero results.

Splunk recommends you avoid this scenario by using reporting commands in the base search.

Avoid returning large numbers of rows in the base search

Passing a large number of search results from a data cube to a post process search can cause problems.

Server time out

If the post-processing operation takes too long, it can result in performance problems, and possibly a timeout due to an unresponsive splunkd daemon/service. In this scenario, consider the following:

  • The number of results and fields returned from the base search.
  • The complexity of the post process operations on these results.

Incomplete data

If the base search is a non-transforming search that returns in excess of the 500,000 event limitation, an incomplete data set is passed to downstream panels (as described above). Construct your data cube with reporting commands in the base search in a way that avoids the 500,000 event limitation.

Display results of a post process search

Modules that support the display of results from a post process search:

  • SingleValue
  • SimpleResultsTable
  • EventsViewer
  • JSChart
  • FlashChart

Modules not supported for display of post process results:

  • MultiFieldViewer
  • ResultsHeader
  • SimpleResultsHeader
  • FlashTimeline
  • SuggestedFieldViewer.

Post process search examples

Post process works best when you reformat results from a base search that uses reporting commands.

This means you can create tables and charts according to specific criteria. For example, you can create different visualizations and reports from the same data set. You can also do further aggregation on the original report.

Basic post process example

This basic example uses reporting commands for the base search, post-processing the results differently:

Base search (data cube search)
index=_internal source=*splunkd.log | stats count by component, log_level

Post process 1
event count by log_level: | stats sum(count) AS count by log_level

Post process 2
error count by component: | search log_level=error | stats sum(count) AS count by component

Complex post process example

For more complex base searches that include statistical aggregations such as percentiles, standard deviations, and even averages, it is better to use summary indexing commands in the base search. This facilitates building the post process searches. Some examples of summary indexing search commands are:

  • sistats
  • sitimechart
  • sitop
  • sichart
  • sirare

The summary index equivalents provide more flexibility for post process searches. For more information see Use summary indexing for increased reporting efficiency and About reporting commands.

Base search (data cube search)
index=_internal | eval event_size=len(_raw)
| sistats count min(event_size) avg(event_size) max(event_size) by source sourcetype

Post process 1
| stats count

Post process 2
| stats avg(event_size) by sourcetype

Post process 3
| stats count by sourcetype

The base search above reports event size (min, avg, max) by source and sourcetype for the _internal index. The sistats count with the various group-by clauses is important. Without these specified in the search you lose the benefits of map-reduce in distributed search.

Caution: When you build your base search, it is tempting to build a simple search that feeds raw events to the post process searches in downstream panels. However, this can be problematic, as described in Post process limitations. The sheer number of raw events in this type of search can easily surpass the 500,000 row limit that can be passed to a post process search. This can result in incomplete results passed to the post process searches.

It is also tempting to build a base search that returns an overwhelming number of rows and fields. This can cause the server to time out during the post process search. Be careful when constructing the base search and the complexity of operations during post process.

Sample code for post process

This section walks you through the advanced XML code to implement the complex post process example.

Add chrome

First, add the chrome and nav for your view:

<view template="dashboard.html">
  <label>Post process examples</label>
  <module name="AccountBar" layoutPanel="appHeader"/>
  <module name="AppBar" layoutPanel="navigationHeader"/>
  
  <module name="Message" layoutPanel="messaging">
    <param name="filter">*</param>
    <param name="clearOnJobDispatch">False</param>
    <param name="maxSize">1</param>
  </module>
  
  <module name="TitleBar" layoutPanel="viewHeader">
    <param name="actionsMenuFilter">dashboard</param>
  </module>
  . . .
</view>

Add the base search

Use the HiddenSearch or HiddenSavedSearch modules to specify the base search.

. . .
<module name="HiddenSearch" layoutPanel="panel_row2_col1" autoRun="True">
 <param name="search">
  index=_internal | eval event_size=len(_raw) 
  | sistats count min(event_size) avg(event_size) max(event_size) by source sourcetype
 </param>
  . . .
 <!-- Add post process modules -->
  . . .
</module>

Post process a search

Use the HiddenPostProcess module to process the results from your base search and feed into a results module. For example, this panel displays search results in a SingleValue module:

<module name="HiddenPostProcess" layoutPanel="panel_row1_col1"
      group="Post process as single value">
  
  <param name="search">
    | stats count
  </param>

  <module name="SingleValue">
    <param name="field">count</param>
    <param name="afterLabel"> events</param>
    <param name="classField">range</param>
  </module>
  
</module>

This panel displays maximum event size by source in a bar chart:

<module name="HiddenPostProcess"
        layoutPanel="panel_row1_col2" group="Post process as bar chart">
  
    <param name="search">
      | stats avg(event_size) by sourcetype
    </param>
  
    <module name="HiddenChartFormatter">
      <param name="chart">bar</param>
      <param name="primaryAxisTitle.text">Source type</param>
      <param name="secondaryAxisTitle.text">Average event size</param>
      <param name="legend.placement">none</param>
      
      <module name="JSChart">
        <param name="width">100%</param>
        <param name="height">200px</param>
      </module>
        
    </module>        
</module>

This panel displays event count per sourcetype in a pie chart:

<module name="HiddenPostProcess"
        layoutPanel="panel_row1_col3" group="Post process as pie chart">

    <param name="search">
      | stats count by sourcetype
    </param>
  
    <module name="HiddenChartFormatter">
      <param name="chart">pie</param>
      <param name="chartTitle">Event count by sourcetype</param>
      
      <module name="JSChart">
        <param name="width">100%</param>
        <param name="height">200px</param>
      </module>
        
    </module>   
</module>

Complete example source code

Here is the complete Advanced XML source code for the dashboard with the three panels shown above.


<view template="dashboard.html">
  <label>Post process examples</label>
  <module name="AccountBar" layoutPanel="appHeader"/>
  <module name="AppBar" layoutPanel="navigationHeader"/>
  
  <module name="Message" layoutPanel="messaging">
    <param name="filter">*</param>
    <param name="clearOnJobDispatch">False</param>
    <param name="maxSize">1</param>
  </module>
  
  <module name="TitleBar" layoutPanel="viewHeader">
    <param name="actionsMenuFilter">dashboard</param>
  </module>

<module name="HiddenSearch" layoutPanel="panel_row2_col1" autoRun="True">
 <param name="search">
  index=_internal | eval event_size=len(_raw) 
  | sistats count min(event_size) avg(event_size) max(event_size) by source sourcetype
 </param>
<module name="HiddenPostProcess" layoutPanel="panel_row1_col1"
      group="Post process as single value">
  
  <param name="search">
    | stats count
  </param>

  <module name="SingleValue">
    <param name="field">count</param>
    <param name="afterLabel"> events</param>
    <param name="classField">range</param>
  </module>
  
</module>

<module name="HiddenPostProcess"
        layoutPanel="panel_row1_col2" group="Post process as bar chart">
  
    <param name="search">
      | stats avg(event_size) by sourcetype
    </param>
  
    <module name="HiddenChartFormatter">
      <param name="chart">bar</param>
      <param name="primaryAxisTitle.text">Source type</param>
      <param name="secondaryAxisTitle.text">Average event size</param>
      <param name="legend.placement">none</param>
      
      <module name="JSChart">
        <param name="width">100%</param>
        <param name="height">200px</param>
      </module>
        
    </module>        
</module>

<module name="HiddenPostProcess"
        layoutPanel="panel_row1_col3" group="Post process as pie chart">

    <param name="search">
      | stats count by sourcetype
    </param>
  
    <module name="HiddenChartFormatter">
      <param name="chart">pie</param>
      <param name="chartTitle">Event count by sourcetype</param>
      
      <module name="JSChart">
        <param name="width">100%</param>
        <param name="height">200px</param>
      </module>
        
    </module>   
</module>
</module>

</view>


Example dashboard with post process searches

The following dashboard shows the results of the post process searches listed above.

Dashboard with post process searches

PREVIOUS
Use lookups with a view
  NEXT
Customization options

This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters