Components and roles
Each segment of the data pipeline directly corresponds to a role that one or more Splunk components can perform. For instance, data input is a Splunk role. Either an indexer or a forwarder can perform the data input role. For more information on the data pipeline, look here.
How components support the data pipeline
This table correlates the pipeline segments and Splunk roles with the components that can perform them:
|Data pipeline segment||Splunk role||Splunk components that can perform this role|
|Data input||Data input|| indexer|
|n/a||Managing distributed updates||deployment server|
|n/a||Troubleshooting deployments||The deployment monitor app (not actually a component, but rather a key feature for managing distributed environments)|
As you can see, some roles can be filled by diffferent components depending on the situation. For instance, data input can be handled by an indexer in single-machine deployments, or by a forwarder in larger deployments.
For more information on components, look here.
Components in action
These are some of the common ways in which Splunk functionality is distributed and managed.
Forward data to an indexer
In this deployment scenario, forwarders handle data input, collecting data and send it on to a Splunk indexer. Forwarders come in two flavors:
- Universal forwarders. These maintain a small footprint on their host machine. They perform minimal processing on the incoming data streams before forwarding them on to an indexer, also known as the receiver.
- Heavy forwarders. These retain much of the functionality of a full Splunk instance. They can parse data before forwarding it to the receiving indexer. (See "How data moves through Splunk" for the distinction between parsing and indexing.)
Both types of forwarders tag data with metadata such as host, source, and source type, before forwarding it on to the indexer.
Note: There is also a third type of forwarder, the light forwarder. The light forwarder is essentially obsolete, having being replaced in release 4.2 by the universal forwarder, which provides similar functionality in a smaller footprint.
Forwarders allow you to use resources efficiently while processing large quantities or disparate types of data. They also enable a number of interesting deployment topologies, by offering capabilities for load balancing, data filtering, and routing.
For an extended discussion of forwarders, including configuration and detailed use cases, see "About forwarding and receiving".
Search across multiple indexers
In distributed search, Splunk instances send search requests to other Splunk instances and merge the results back to the user. This is useful for a number of purposes, including horizontal scaling, access control, and managing geo-dispersed data.
The Splunk instance that manages search requests is called the search head. The instances that maintain the indexes and perform the actual searching are indexers, called search peers in this context.
For an extended discussion of distributed search, including configuration and detailed use cases, see "About distributed search".
Manage distributed updates
When dealing with distributed deployments consisting potentially of many forwarders, indexers, and search heads, the Splunk deployment server simplifies the process of configuring and updating Splunk components, mainly forwarders and indexers. Using the deployment server, you can group the components (referred to as deployment clients in this context) into server classes, making it possible to push updates based on common characteristics.
A server class is a set of Splunk instances that share configurations. Server classes are typically grouped by OS, machine type, application area, location, or other useful criteria. A single deployment client can belong to multiple server classes, so a Linux universal forwarder residing in the UK, for example, might belong to a Linux server class and a UK server class, and receive configuration settings appropriate to each.
For an extended discussion of deployment management, see "About deployment server".
View and troubleshoot your deployment
Use the deployment monitor app to view the status of your Splunk components and troubleshoot them. The deployment monitor is functionally part of the search role, so it resides with either the indexer or a search head. It looks at internal events generated by Splunk forwarders and indexers.
The home page for this app is a dashboard that provides charts with basic stats on index throughput and forwarder connections over time. It also includes warnings for unusual conditions, such as forwarders that appear to be missing from the system or indexers that aren't currently indexing any data.
The charts, warnings, and other information on this page provide an easy way to monitor potentially serious conditions. The page itself provides guidance on what each type of warning means.
The deployment monitor also provides pages that consolidate data for all forwarders and indexers in your deployment. You can drilldown further, to obtain detailed information on any forwarder or indexer.
For more information on the deployment monitor, read the Deploy and Use Splunk Deployment Monitor App manual.
For more information
In summary, these are the fundamental components and features of a Splunk distributed environment:
- Indexers. See "About indexes and indexers" in the Managing Indexers and Clusters manual.
- Forwarders. See "About forwarding and receiving" in this manual.
- Search heads. See "About distributed search" in this manual.
- Deployment server. See "About deployment server" in this manual.
- License master. See "Configure a license master".
- Cluster master. If a replication cluster is being used. See "About clusters and indexing".
For guidance on where to configure various Splunk settings, see "Configuration parameters and the data pipeline" in the Admin manual. That topic lists key configuration settings and the data pipeline segments they act upon. If you know which components in your Splunk topology handle which segments of the data pipeline, you can use that topic to determine where to configure the various settings. For example, if you use a search head to handle the search segment, you'll need to configure any search-related settings on the search head and not on your indexers.
Scale your deployment: Splunk components
Hardware capacity planning for a distributed Splunk Enterprise deployment
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18