Chain searches together with a base search and chain searches

When you use a separate search for each visualization on a large dashboard, you can use a lot of computing power. When these searches begin with the same initial SPL search sections, you can use these sections as a base search and extend it using additional, chained data source searches that will drive the same visualizations, using less computing power because the base search is only run one time for all of the visualizations. This can be done directly in the visual editor.

For example, three data sources have the following three searches which begin with the same first two search sections:
Search 1

index=_internal | stats count by status | where status < 300 | stats sum(count) as "Success"

Search 2

index=_internal | stats count by status | where status >= 400 | stats sum(count) as "Failed"

Search 3

index=_internal | stats count by status | where status >= 400 | where status < 500 | stats sum(count) as "UserError"

You can create two data sources of type ds.chain that tie them to an existing base search. For example:

Chain search 1

| where status < 300 | stats sum(count) as "Success"

Chain search 2

| where status >= 400

You can further extend a chain search by one additional search. For example:

Chain search 2a

| stats sum(count) as "Failed"

Chain search 2b

| where status < 500 | stats sum(count) as "UserError"

Create a chain search

1. Select a visualization and click Set up a Primary data Source + in the Data Configurations section of the Configuration panel.
2. Select Create Search to create your base search.
3. Select a second visualization and select Create a Chain Search.
4. Use the drop down menu under Parent Search to find and select your base search.
5. Add the SPL to create the chain search.

You can use this method to create as many chain searches off of the base search as you want, but you can only create one additional chain search by extending an existing chain search. To extend a chain search, choose the first chain search as the parent search instead of selecting your base search.

Example

There are many different combinations you can use once you've established the base search, for example:

The base search becomes:

index=_internal | stats count by status

Search 1 is now the following: base search + Chain search 1

Search 2 is now the following: base search + Chain search 2 + Chain search 2a

Search 3 is now the following: base search + Chain search 2 + Chain search 2b

You can extend many independent chain searches from the base search, and you can extend many second level chains that use the first level chain as their primary data source, but you cannot have a third level of chains that use the a second level chain as a primary data source.

This functionality is similar to the way that you might have used post-process searches using Simple XML.

You can use tokens in both base and chain searches. Any token you create can be used in a search of type ds.chain, but time-related tokens can only be used in the base search.

You cannot specify any properties such as queryParameters, refresh , and refreshType for chain searches. These are inherited from those set in the base search, or from settings in the defaults section. When a base search is refreshed, or its SPL search is changed, the associated chain searches will also be refreshed.

An example of the anatomy of the chain search

In the following example, there are four searches. One base search and three chain searches. One chain search relies on the base search, and the other two rely on the first chain search.

ds.chain example

Expand this window to see the dashboard definition of the example.

{
  "visualizations": {
    "viz_2AFU2XSx": {
      "type": "splunk.column",
      "dataSources": {
        "primary": "ds_g9BqIN8h"
      },
      "title": "Base Data",
      "options": {
        "yAxisAbbreviation": "off",
        "y2AxisAbbreviation": "off",
        "showRoundedY2AxisLabels": false,
        "legendTruncation": "ellipsisMiddle",
        "showY2MajorGridLines": true
      },
      "context": {}
    },
    "viz_9zvXENZu": {
      "type": "splunk.table",
      "dataSources": {
        "primary": "ds_OSNBwGNz"
      },
      "title": "Chained Data",
      "options": {
        "count": 20,
        "tableFormat": {
          "data": "> table | formatByType(formattedConfig)"
        }
      },
      "context": {
        "formattedConfig": {
          "number": {
            "precision": 0,
            "thousandSeparated": false,
            "unitPosition": "after"
          },
          "string": {
            "unitPosition": "after"
          }
        }
      }
    },
    "viz_61GNJM6Y": {
      "type": "splunk.singlevalue",
      "title": "Average",
      "dataSources": {
        "primary": "ds_Ah56LDFf"
      },
      "context": {},
      "options": {
        "showSparklineAreaGraph": false,
        "sparklineValues": "> primary | seriesByTypes(\"number\", \"string\")"
      }
    },
    "viz_gnZcFdBS": {
      "type": "splunk.singlevalue",
      "title": "Count",
      "dataSources": {
        "primary": "ds_dFHviA1E"
      },
      "context": {},
      "options": {
        "showSparklineAreaGraph": false,
        "sparklineValues": "> primary | seriesByTypes(\"number\", \"string\")"
      }
    },
    "viz_UZfgjCjy": {
      "type": "abslayout.line",
      "options": {}
    },
    "viz_aGYxaiBz": {
      "type": "abslayout.line",
      "options": {}
    },
    "viz_asyqhgft": {
      "type": "abslayout.line",
      "options": {}
    }
  },
  "dataSources": {
    "ds_g9BqIN8h": {
      "type": "ds.search",
      "options": {
        "query": "index=_internal \n|  top 100 sourcetype"
      },
      "name": "Base search"
    },
    "ds_OSNBwGNz": {
      "type": "ds.chain",
      "options": {
        "query": "| where count > 10",
        "extend": "ds_g9BqIN8h"
      },
      "name": "Chain search 1 extends base search"
    },
    "ds_Ah56LDFf": {
      "type": "ds.chain",
      "options": {
        "query": "| stats avg",
        "extend": "ds_OSNBwGNz"
      },
      "name": "Chain search 2 extends chain search 1"
    },
    "ds_dFHviA1E": {
      "type": "ds.chain",
      "options": {
        "query": "|  stats count",
        "extend": "ds_OSNBwGNz"
      },
      "name": "Chain search 3 extends chain search 2"
    }
  },
  "defaults": {},
  "layout": {
    "type": "absolute",
    "options": {},
    "structure": [
      {
        "item": "viz_2AFU2XSx",
        "type": "block",
        "position": {
          "x": 20,
          "y": 30,
          "w": 350,
          "h": 290
        }
      },
      {
        "item": "viz_9zvXENZu",
        "type": "block",
        "position": {
          "x": 420,
          "y": 20,
          "w": 440,
          "h": 310
        }
      },
      {
        "item": "viz_61GNJM6Y",
        "type": "block",
        "position": {
          "x": 940,
          "y": 20,
          "w": 150,
          "h": 150
        }
      },
      {
        "item": "viz_gnZcFdBS",
        "type": "block",
        "position": {
          "x": 940,
          "y": 200,
          "w": 150,
          "h": 150
        }
      },
      {
        "item": "viz_UZfgjCjy",
        "type": "line",
        "position": {
          "from": {
            "item": "viz_2AFU2XSx",
            "port": "e"
          },
          "to": {
            "item": "viz_9zvXENZu",
            "port": "w"
          }
        }
      },
      {
        "item": "viz_aGYxaiBz",
        "type": "line",
        "position": {
          "from": {
            "item": "viz_9zvXENZu",
            "port": "e"
          },
          "to": {
            "item": "viz_61GNJM6Y",
            "port": "w"
          }
        }
      },
      {
        "item": "viz_asyqhgft",
        "type": "line",
        "position": {
          "from": {
            "item": "viz_9zvXENZu",
            "port": "e"
          },
          "to": {
            "item": "viz_gnZcFdBS",
            "port": "w"
          }
        }
      }
    ]
  },
  "description": "",
  "title": "Base and Post Search Example"
}

If you are using lookups in your search, make sure the lookup permissions are set to "global".

Best practices for creating chain searches

Use these best practices to make sure that chain searches work as expected.

Use a transforming base search

A base search should be a transforming search that returns results formatted as a statistics table. For example, searches using the following commands are transforming searches: stats, chart, timechart, and geostats, among others. For more information on transforming commands, see About transforming commands in the Search Manual.

About search result and timeout issues

Non-transforming base searches can cause the following search result and timeout issues. If you observe these issues in a dashboard, check the base search to make sure that it is a transforming search.

No results returned
If the base search is a non-transforming search, you must explicitly state in the base search what fields will be used in the chain search using the | fields command. For example, if your chain search will search for the top selling buttercup game categories over time, you would use a search command similar to the following.

| fields _time, categoryId, action

Event retention
If the base search is a non-transforming search, the Splunk platform retains only the first 500,000 events that it returns. A chain search does not process events in excess of this 500,000 event limit, silently ignoring them. This can generate incomplete data for the chain search.

This search result retention limit matches the max_count setting in limits.conf. The setting default is 500,000.

Client timeout
If the chain search takes too long, it can exceed the Splunk Web client timeout value of 30 seconds.

The collect command

The collect command does not work with chain searches when used in the base search.

For more information about transforming searches, see transforming commands and searches in the Search Manual.

Do not reference fields in chain searches that are not referenced in the base search

A chain search depends entirely on the fields present in the base search. If you are not referencing a particular field in the base search, do not reference it in the chain search. Fields used in transforming commands will automatically be available for chain searches. When transforming commands are not used in a base search, fields without a reference in the base search appear null in a chain search. The chain search returns no results in this case.

Limit base search results and chain complexity

Passing a large number of search results to a chain search can cause server timeout issues. In this scenario, consider adjusting the base search to reduce the number of results and fields that it returns. You can also consider reducing the complexity of chain searches on the base search results.

You can use a single chain search from a base search to generate results or you can generate multiple chain searches together.