Examine configuration files to determine your topology

In this method of discovery, you examine certain configuration files residing on each Splunk Enterprise instance. The files contain settings whose presence or absence help you to determine what component the instance functions as. The settings also help determine the relationships between components, and thus, the overall topology.

The discovery process looks for the characteristic configurations of each component type.

It can be helpful to draw a diagram of your deployment, as you go about the discovery process. See Draw a diagram of your deployment.

Prerequisites

Read the following material:

• The chapter Administer Splunk Enterprise with configuration files in the Admin Manual. The topics in this chapter provide important background information on configuration files: what they are, where they reside, and how they layer on top of one another.

• Deployment topologies. This topic describes the elements of Splunk Enterprise deployments and offers essential guidance on how to discover your deployment topology.

Configuration file locations

Copies of configuration files can reside in several locations, including system directories and app directories. If a configuration file has copies in multiple locations, the operative value of each setting in the file is determined by a process of file layering, based on an order of precedence.

For details on configuration file locations, see Configuration file directories in the Admin Manual. To learn how multiple copies of a configuration file layer, see Configuration file precedence in the Admin Manual.

The files that contain component configurations, such as server.conf, usually reside only under $SPLUNK_HOME/etc/system/local, because component settings are system-level configurations and should not be app-dependent. To be certain of your configurations, however, you must examine all possible locations for copies of each relevant file.

How to examine configuration files

To examine a configuration file, you can use either a text editor or the btool utility.

Caution: If you use a text editor, do not make any changes to the files.

The btool utility in $SPLUNK_HOME/bin provides a quick way to sift through all copies of a configuration file. The advantage of btool is that it reports back the final set of configurations that results after layering all copies of the file. For syntax and other details, see Use btool to troubleshoot configurations in the Troubleshooting Manual.

Discover processing components

First, examine each instance to see if it hosts a processing component. Later, you examine the instances to see if they host management components.

To discover the processing components, follow these two procedures:

1. Identify search heads and indexers.
2. Identify forwarders.

An instance typically contains at most one processing component. Therefore, for each instance, follow each procedure, in order, until you identify what processing component, if any, the instance contains.

Processing components in the procedures are identified by bold text.

Identify search heads and indexers

Determine whether the instance is a search head or an indexer and, if so, what type of search head or indexer.

1. Examine server.conf in $SPLUNK_HOME/etc/system/local.
   1. Look for a [clustering] stanza. If found, this instance is a node of an indexer cluster.
      
      To determine the indexer cluster node type, examine the mode setting:
      1. If mode = master or mode = manager, this instance is an indexer cluster manager node. Stop here.
         
         Note: The manager node is a management component, not a processing component. Because it is configured in server.conf, however, it makes sense to look for it while examining server.conf for processing components.
      2. If mode = slave or mode = peer, this instance is an indexer cluster peer node, also known as a clustered indexer. Stop here.
      3. If mode = searchhead , this instance is an indexer cluster search head node. Continue to the next step, to determine whether this indexer cluster search head is also a member of a search head cluster.
   2. Look for a [shclustering] stanza. If found, this instance is a search head cluster member. Stop here.
      
      Note: Depending on the deployment topology, a search head cluster member can also be a search head node in an indexer cluster.
   3. Look for a [pooling] stanza. If found, this instance is a a search head pool member. Stop here.
2. Examine distsearch.conf in $SPLUNK_HOME/etc/system/local.
   1. Look for a populated servers setting in the [distributedSearch] stanza. If found, this instance is an independent search head. Stop here.
      
      Note: Search heads that are members of search head clusters or search head pools also have a populated servers setting. However, you have already identified such search heads in earlier steps of this procedure, so any search heads with a populated servers setting that remain to be identified at this point of the procedure must be independent search heads.
3. On all instances previously identified as either search head cluster members, search head pool members, or independent search heads, examine the servers setting in distsearch.conf.
   
   The list of addresses in servers specifies the hosts for the indexers that this search head connects to. Use this list to determine which instances in your deployment are independent (non-clustered) indexers. Stop here.
   
   Note: In an indexer cluster, search heads do not use the servers list to specify their indexers. Therefore, if an indexer appears in this list, it is a non-clustered indexer.

Once you have identified your search heads and indexers, any remaining instances are either forwarders or management components.

The following flowchart encapsulates the procedure above:

Caution: In complex deployments, a search head might manage searches for multiple indexer clusters, or for both an indexer cluster and a set of independent indexers. If you suspect that you have such a deployment topology, you can investigate the search head's configuration more deeply. See Search across multiple indexer clusters and Search across both clustered and non-clustered search peers in Managing Indexers and Clusters of Indexers.

Identify forwarders

If the instance is not an indexer or a search head, it might be a forwarder.

The indexers in a Splunk deployment provide information on the forwarders that have been installed in that deployment. Indexers in a deployment keep logs of each forwarder connection and type.

Use Splunk search to identify forwarders

You can get a list of every forwarder that has connected to a Splunk indexer or indexer cluster by logging into Splunk Enterprise on an indexer or search head and running the following search in Splunk Web:

index=_internal source=*metrics.log group=tcpin_connections | where isnotnull(fwdType) | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | dedup sourceHost | eval connectionType =case(fwdType=="uf","Universal", fwdType=="lwf", "Lightweight", fwdType=="full","Heavy") | rename sourceIp as "Source IP", sourceHost as "Source Host", connectionType as "Forwarder Type" | table "Source IP" "Source Host" "Forwarder Type"

If you have the IP address or host name of a machine whose forwarder status and type you want to check, you can modify the search to specify the host name or IP address:

index=_internal source=*metrics.log group=tcpin_connections [sourceIp=<ip address>|hostname=<host name>] | where isnotnull(fwdType) | eval sourceHost=if(isnull(hostname), sourceHost,hostname) | dedup sourceHost | eval connectionType =case(fwdType=="uf","Universal", fwdType=="lwf", "Lightweight", fwdType=="full", "Heavy") | rename sourceIp as "Source IP", sourceHost as "Source Host", connectionType as "Forwarder Type" | table "Source IP" "Source Host" "Forwarder Type"

If the table that these searches generate does not list information on the machine you want information about, or you do not have access to Splunk Web, you can examine the machine itself to determine the forwarder type.

Manually identify forwarders

1. On the machine whose status you want to check, look in the $SPLUNK_HOME/bin directory. Check to see if a python executable (python.exe on Windows) is present in this directory.
1. If there is not, then this instance is a universal forwarder. You can stop here or proceed to Identify intermediate forwarders in the next section.
2. If there is, then review $SPLUNK_HOME/etc/apps/SplunkLightForwarder/local/app.conf to see if the state setting in that file is set to enabled.
1. If it is, then the instance is a light forwarder. Proceed to Identify intermediate forwarders in the next section in either case.
2. If it is not, then the instance is a heavy forwarder. Proceed to Identify intermediate forwarders in the next section in either case.

Identify intermediate forwarders

Intermediate forwarders receive data from other forwarders to send to indexers. Any forwarder can be an intermediate forwarder.

1. Review $SPLUNK_HOME/etc/system/local/inputs.conf on the forwarder. Is there a splunktcp:// stanza present and enabled?
• If there is, then the instance is an intermediate forwarder.
• If there is not, then the instance is not an intermediate forwarder.

See the following flowchart to understand how forwarder discovery works:

From processing components to management components

Once you reach this point for all your Splunk Enterprise instances, you know all your processing components, as well as the indexer cluster manager management component, if any.

If you cannot find any processing component on a particular instance, the instance hosts management components only.

In the next section, determine which instances host management components.

Discover management components

Management components can be located on their own dedicated Splunk Enterprise instance, co-located with a processing component, or co-located with other management components. Therefore, any of your Splunk Enterprise instances could contain management components.

Universal forwarder instances, however, cannot also contain management components.

Also, there is usually at most one of each management component for an entire deployment So, for example, once you identify the instance that hosts the deployment server, if any, you can stop looking for other deployment server instances.

This table shows, for each management component, its key indicator, along with the configuration file that pertains to the indicator, if any.

Component type	Configuration file	Key indicator
Monitoring console	$SPLUNK_HOME/etc/apps/splunk_monitoring_console /splunk_monitoring_console_assets.conf	the existence of splunk_monitoring_console_assets.conf
Deployment server	$SPLUNK_HOME/etc/system/local/serverclass.conf	[serverClass:<name>] stanzas
License manager	$SPLUNK_HOME/etc/system/local/server.conf	[license] stanza with manager_uri = self or master_uri = self
Indexer cluster manager	$SPLUNK_HOME/etc/system/local/server.conf	[clustering] stanza with mode = manager or mode = master
Search head cluster deployer	N/A	populated $SPLUNK_HOME/etc/shcluster/ directory

If you have completed all the steps in this topic, you should now understand the specifics of your Splunk Enterprise topology. You should know the location and function of each instance and the relationships between instances.