Set up your system for the Splunk Add-on for VMware

Plan your installation in a test environment

Install the Splunk Add-on for VMware and its prerequisites into a test environment before you install it in a production environment. This way you can work out any issues that you might encounter in your deployment.

After you install the Splunk Add-on for VMware and its prerequisites in your test environment, scale the deployment with more advanced Splunk platform deployment features, such as search head clustering and indexer clustering.

If you don't have access to a test environment, limit the number of hosts and vCenter Servers you use when you first deploy the app and then add complexity after your initial setup is successful.

Splunk Add-on for VMware sample test environment

Use the following test environment size:

• One vCenter Server that supports 40 or fewer ESXi hosts.
• One instance of Splunk Enterprise with one search head and one indexer. See Platform and hardware requirements for Splunk Enterprise versions that support the Splunk Add-on for VMware.
• One Data Collection Node (DCN).

See Install the Splunk OVA for VMware Metrics in your virtual environment section in Splunk OVA for VMware Metrics for DCN system requirements.

Create your data collection nodes

Data Collection Nodes (DCNs) are custom Splunk forwarders for connecting, polling data from and enriching data from your VMWare vCenters and forwarding it back to your indexers. See Install the Splunk OVA for VMware Metrics in your virtual environment for more information.

In your test environment, deploy the DCN using the configured Splunk OVA to collect vCenter Server API data. With the following specifications, one data collection node can collect from 40 ESXi hosts or fewer, with a ratio of 25 to 30 virtual machines per host. The default virtual machine included with the Splunk Add-on for VMware is set with this configuration.

• Four cores. Four vCPUs or two vCPUs with two cores with a reservation of 2GHz
• 6GB memory with a reservation of 1GB
• 10-12GB of disk space

Optionally, you can set up a syslog collector when you install and configure Splunk Add-on for VMware. This action is not required for a working VMware app deployment. See the Configure the Splunk Add-on for VMware to collect data section of the Splunk Add-on for VMware to learn more about syslog collection.

See the Deploy OVA to create a Data Collection Node section of the Splunk OVA for VMware and NetApp to learn more.

Set up a vCenter Server user account

Obtain VMware vCenter Server account credentials for each vCenter Server system.

These credentials allow the Splunk Add-on for VMware read-only API access to the appropriate metrics on each vCenter Server system in the environment. the Splunk Add-on for VMware uses the credentials when the DCN polls vCenter Server systems for performance, hierarchy, inventory, task, and event data. These credentials are required for DCN configuration. You can use existing vCenter Server account credentials, or create a new account for Splunk Add-on for VMware to access the vCenter Server data.

Permissions in vSphere

Splunk Add-on for VMware must use valid vCenter Server service credentials to gain read-only access to vCenter Server systems using API calls. The account's vSphere role determines access privileges.

The following sections list the permissions for the vCenter server roles for all of the VMware versions that Splunk App for VMware supports.

Permissions to use your own syslog server

Best practice dictates that use your own syslog server, and that you install a Splunk Enterprise forwarder on the server to forward syslog data. Use these permissions to collect data from the ESXi hosts using your own syslog server. These system-defined privileges are always present for user-defined roles.

Permission
System.Anonymous
System.Read
System.View

Permissions to use an intermediate forwarder

Use these permissions if you configure your ESXi hosts to forward syslog data to one or more intermediate Splunk Enterprise forwarders. Use the vSphere Client to enable the syslog firewall for the specific hosts. For vSphere version 7.0, you don't need to add permissions beyond the default permissions that vSphere provides when creating a role.

Permission
System.Anonymous
System.Read
System.View
Host.Config.AdvancedConfig

Validate vCenter Servers time synchronization settings

Verify time synchronization throughout your environment to improve visibility into application and operating system health. Check the time synchronization for the following components in your environment.

• Hosts
• Splunk Enterprise search head and indexers

Consider using NTP or VMware host/guest time synchronization.

Configure ports

Collect data from vCenter Server systems using the VMware API

The Splunk Add-on for VMware and its prerequisite add-ons use the VMware API to collect data about your virtual environment. The Splunk Add-on for VMware and its prerequisite add-ons communicate with vCenter Server using network ports and Splunk management ports.

This table lists the components that communicate with each other and the ports they use to communicate.

Sender	Receiver	Port number	Description
Collection Configuration	vCenter server	443	Uses port 443 to connect to the vCenter Server to verify that the vCenter Server credentials are valid. It uses this port to discover the number of managed ESXi hosts in the environment.
Splunk Add-on for VMware	Data Collection Node	8089	Connects to the Data Collection Node (DCN) on the default Splunk management port, TCP 8089.
Collection Configuration	Data Collection Node	8008	When the DCN and Splunk Add-on for VMware have established a connection, the Collection Configuration dashboard, which typically runs on the search head, allocates data collection jobs to the DCN on the TCP port 8008 (gateway port). In your environment, if another service uses port 8008, you can configure a different port for communication between the data collection node and the gateway. Data collection nodes do not have to communicate on the same port. [default] gateway_port = 8008 To change the ports for each data collection node individually, set the port in each stanza.
Data Collection Node (DCN)	vCenter Server	443	Communicates with vCenter Server API on port 443 to execute the data collection tasks allocated to it.
Data Collection Node	Splunk indexer	9997	Uses port 9997 to forward data it has retrieved from the vCenter Server using the API.

After the Splunk Add-on for VMware establishes a connection with vCenter Server, the DCN uses port 443 to obtain the credentials for vCenter Server. The DCN uses port 443 to determine the kind of data to collect, such as performance, inventory, or hierarchy data. The Splunk Add-on for VMware sends information to the data collection nodes using port 8008 about the information they need to collect from a specific vCenter Server system. The DCN retrieves the data from vCenter Server and forwards the data to the Splunk indexer on port 9997.

Collect log data from vCenter Server systems and ESXi hosts

You can collect log data from the vCenter Server system and the ESXi hosts in your environment. This table describes how the entities in your environment communicate.

Sender	Receiver	Port number	Description
vCenter server	Splunk indexer	9997	To send log data from the vCenter Server system on port 9997, install the Splunk Universal Forwarder and the Splunk_TA_vcenter on the vCenter Server system. If firewall issues prevent you from installing the Splunk Add-on for vCenter Logs (Splunk_TA_vcenter) components on vCenter Server, forward the vCenter Server log data to the data collection node (DCN). The DCN contains all of the components required to collect vCenter Server log data. Forward this data from the DCN to Splunk indexers.
ESXi host	DCN/ Syslog server	TCP port 1514 / UDP port 514	Prior to ESXi version 6.x, ESXi versions supported either TCP or UDP, but not always both. For an environment with fewer than 40 ESXi hosts, send syslog traffic to the Data Collection Scheduler (DCS), which controls the collection by DCNs. In a larger production environment, use a central syslog server with a Splunk Universal Forwarder and Splunk_TA_esxilogs add-on installed on it. Alternatively, you can send syslog to another DCN virtual machine dedicated to run as a syslog server for the ESXi hosts.
vCenter Servert	DCN/ Syslog server	TCP Port 1517	To send log data from vCenter Linux Server on port 1517 use Syslog-ng/rsyslog. See Collect vCenter Server Appliance logs via syslog.

Prepare to host a data collection node

The Splunk Add-on for VMware uses a virtual appliance version of the Data Collection Node (DCN) to collect performance metrics. Splunk distributes this as an Open Virtual Appliance (OVA) file called the Splunk OVA for VMware metrics.

Splunk configures the DCN with the following default configuration:

• Eight cores. 8 vCPUs or 4 vCPUs with two cores with a reservation of 2GHz.
• 12GB memory with a reservation of 1GB.
• 16GB of disk space.

In production, the DCNs communicate with the Collection Configuration dashboard, which runs on the Splunk search head, to retrieve data from vCenter Server. To ensure reliable communication between systems, use static IP addresses and dedicated host names for each DCN. See Collect Data from vCenter Server systems using the VMware API.

Prepare to deploy the DCN

• Identify the vCenter servers and managed ESXi hosts from which you want to collect data.
• Determine the number of DCNs that you want to deploy. Each DCN can collect data from 70 or fewer ESXi hosts, based on the specifications for the 8 core DCN configured with the OVA for VMware with a ratio of 25 to 30 virtual machines per host.
• Each Data Collection Node (DCN) needs at least one CPU core for every 10 hosts from which the DCN is collecting data.
• Estimate the number of CPUs needed for your worker processes with the expectation that a CPU in your deployment can be kept as a spare for other processes. Provision at least one extra CPU to help promote capacity and availability in your deployment.
• Obtain static IP addresses and host names to apply to each of the DCNs.

Identify if a dedicated Distributed Collection Scheduler is needed

A Linux-based dedicated Distributed Collection Scheduler (DCS) is required if any of the following scenarios apply.

• Your search head is running on Windows
• Your search heads are in a search head cluster
• Site-specific collection is desired

If one or more of the cases above is true, you must plan to have an additional Splunk instance running on Linux to perform the collection. (The OVA image above can be used to create this additional instance.)

When you have this information, you can then create the data collection nodes. For more information, see Configure the Splunk OVA for VMware Metrics.