Troubleshooting

Change collection intervals

Change collection intervals to reduce the load on your Data Collection Nodes (DCNs) and your vCenter Servers.

1. Change the time interval for your host inventory job.
2. On the instance where your scheduler is running, navigate to \etc\apps\Splunk_TA_vmware\default\.
3. Open the ta_vmware_collection.conf file.
4. Change hostinv_interval and hostinv_expiration from the 900 second default to a larger number (maximum 2700 seconds). Keep hostinv_interval and hostinv_expiration at the same number of seconds.
5. Save your changes and exit.
6. Change the time interval for host performance data.
7. On the instance where your scheduler is running, navigate to \etc\apps\Splunk_TA_vmware\local\.
8. Open the ta_vmware_collection.conf file.
9. Change hostvmperf_interval and hostvmperf_expiration from the 180 second default to a larger number (maximum 1200 seconds). Keep hostvmperf_interval and hostvmperf_expiration at the same number of seconds.
10. Save your changes and exit.

Change the NTP server pool list

A computer uses the Network Time Protocol (NTP) to synchronize its time with another reference time source. If you are experiencing time synchronization issues between the indexer, DCN, and vCenter Server, you can change the NTP servers that the DCN uses by editing the /etc/ntp.conf file.

1. Access the /etc/ntp.conf file. The following values are defaults for the servers in the file.

 # Use public servers from the pool.ntp.org project.
 # Please consider joining the pool (http://www.pool.ntp.org/join.html).
 server 0.centos.pool.ntp.org
 server 1.centos.pool.ntp.org
 server 2.centos.pool.ntp.org

2. Replace the default values in the file with your NTP server values.
3. Restart ntpd using the following command: sudo service ntpd restart

Disable NTP on the data collection node

If your firewall prevents Internet access, you can disable NTP on the data collection node. If you disable NTP, enable VMware Tools Clock Synchronization which establishes the time for the DCN using the ESXi host.

1. Log in to Splunk Enterprise as a splunk user.

2. Stop the ntpd service using the following command: sudo service ntpd stop

3. Configure ntpd so that it does not run when the system starts: sudo chkconfig ntpd off

4. Enable VMware Tools Clock Synchronization using the following command: vmware-toolbox-cmd timesync enable

5. Confirm that the VMware Tools Clock Synchronization is enabled correctly using the following command: vmware-toolbox-cmd timesync status

vpxd.stats.maxQueryMetrics error prevents data collection from vcenters

As of version 6.0 VMware vCenter has added a limitation to the number of performance metrics collected by the vpxd.stats.maxQueryMetrics function. The vCenter 6.0 maxQuerySize limit is 64 metrics per query. This 64 metric limit is calculated by mulltiplying the number of metrics queried by the number of entities (virtual machines) being queried. For example, querying 8 entities (virtual machines) for 10 metrics from each entity (virtual machine) equals a query size of 80.

To adjust the maxQuerySize limit:

1. Navigate to the advanced settings of vCenter Server, or vCenter Server Appliance.
2. Edit the config.vpxd.stats.maxQueryMetrics key.
3. Edit the web.xml file.

See the VMware documentation for more information.

Remove multiple assigned IP Addresses

If the deployed OVA machine is assigned multiple IPs either by running dcn-network-config script or by other method then the customer can use the below command to remove the IP's. (Note: The below command needs to be run for each IP)
ip addr del <ip_to_be_deleted>/subnet_mask dev <interface_name>
for eg. deleting 192.11.22.33/24 assigned to eth0 interface, the command will look like as below:
ip addr del 192.11.22.33/24 dev eth0

Change the NTP server pool list in RHEL 9

In RHEL 9, we use chrony for NTP. We can change the NTP servers that the DCN uses by editing the /etc/chrony.conf file.

1. Access the /etc/chrony.conf file.

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
 server 1.rhel.pool.ntp.org iburst

2. Replace the default values in the file with your NTP server values.
3. Restart chrony using the following command: sudo systemctl restart chronyd.service