Correlate NetApp and VMware data

Issues in the storage layer can impact the performance of virtual machines in your environment. Using the Splunk App for NetApp Data ONTAP, as an administrator, you can correlate NetApp storage issues with the (VMware) virtual and infrastructure elements. Use the "NFS Volumes Used by VMware" report provided in this app as a starting point to identify the scope of errors in your environment and to better troubleshoot problems in your storage layer. Drill down to the specific virtual machines, storage controllers, and so on that have an impact on the performance of your environment. Volume events provide you with information about the status of your volumes so that you can be proactive in monitoring storage issues.

Correlating NFS-specific storage performance between your VMware and NetApp environments

High latency is an indication of inefficiencies somewhere in your environment. If high latency is a factor in your VMware infrastructure, you can quickly correlate the VMware data with performance information from NetApp controllers in the storage layer. Your environment can be I/O constrained on the ONTAP filers in your NetApp environment or you can have performance issues in your VMware infrastructure. Either way, using the "NFS Volumes Used by VMware report" you can quickly identify the cause of the latency conditions, determine the scope of the condition and identify where the problem exists between the VMware hosts and your NetApp ONTAP filers. You can investigate further to determine if a high latency condition affects all VMware hosts in your environment or only a subset of those hosts, and if it has an impact on the virtual machines on a given datastore or NetApp ONTAP storage controller.

An increase in Network File System (NFS) storage traffic can result from many legitimate process running on many virtual machines at the same time. What you want to identify are:

• the processes that cause increased storage IOPS/throughput in the virtual environment
• the processes that run in the operating systems of our virtual machines that cause an increase in this NFS storage traffic.

Requirements

To run the "NFS Volumes Used by VMware" report, you must have the following apps installed in your environment:

• the Splunk App for VMware
• the Splunk App for NetApp Data ONTAP

Required sourcetypes

The following sourcetypes must be present for this search to populate a dashboard:

• ontap:volume
• vmware:inv:datastore

Understand the NFS Volumes Use by VMware report

The following search powers the "NFS Volumes Use by VMware" report:

sourcetype="ontap:volume" (source=volume-get-iter OR source=volume-list-info-iter-start) | eval name=coalesce(name, "volume-id-attributes.name") | stats values(name) as volname by host | lookup dnslookup clienthost AS host OUTPUT clientip AS ip | mvexpand volname | table * | join type=inner ip, volname [search `VmwareNFSMounts`] | rename name as "Datastore name", path as "Path", volname as Volume, filer as "Filer (VMware data)", host as Filer, ip as IP, vcenter as VCenter

The "ontap:volume" sourcetype must be present to get information about the volumes in the NetApp environment. The name of the volume and ths IP address of the host associated with the volume is returned. `VmwareNFSMounts`is a macro used to search your VMware environment for the NFS volumes used.

The following definition for the `VmwareNFSMounts` macro is used to search the VMware data for NFS data stores:

sourcetype=vmware:inv:datastore changeSet.info.nas.type="NFS" earliest=-24h latest=now | rename changeSet.info.nas.name as name, changeSet.info.nas.remotePath as path, changeSet.info.nas.remoteHost as filer, host as vcenter | dedup path, filer | rex field=path ".+/(?<volname>[^/]+)" | lookup dnslookup clienthost AS filer OUTPUT clientip AS ip | table name, path, filer, ip, volname, vcenter

Go to Settings, then Advanced Search and click Search macros to see the definition for this macro in the Splunk App for NetApp Data ONTAP.

This macro searches the VMware environment for NFS volumes. The initial search results and the results of the macro are joined in a table to include the VMware datastore name and the NetApp ONTAP filer/volume information. The fields of the resulting table are:

• Name - This is the name of the datastore in VMware environment.
• Path - This is the mount path.
• Filer - This is the name of the NetApp filers used as NFS mounts in the VMware environment.
• IP - This is the IP address of the filer (ONTAP server).
• Volname - This is the name of the volume(s) used as the NFS mount(s) in the VMware environment.
• vCenter - This is the virtual center managing your VMware environment.

Using this table you can match the data in the Splunk App for VMware with the data in the Splunk App for NetApp Data ONTAP.

Example of a performance query

As we mentioned previously, performance changes in virtual machines in your environment can be related to issues at the storage level. To create and run a performance query in your environment, you must know the filer hostname and the volume name. Get the volume name from the Splunk App for VMware VM detail view, which lists datastore information for that virtual machine. Get the NetApp volume name and the filer hostname from the "NFS Volumes Used by VMware" report. The following is an example of a query that does this.

For example, if virtual machines on an NFS datastore named "ISO" (in the Splunk App for VMware) cause a problem in your environment, run the following search to get the associated host name and the NetApp volume name:

| savedsearch "NFS Volumes Used by VMware (requires VMware app)" | search "Datastore name"="ISO"

Use the results of this query to develop further searches against the NetApp index. For example, if the above search returns "10.160.114.230" in the Filer field and "testvolume" in the Volume field, you can run the following search to gather performance data for that volume:

index=ontap host="10.160.114.230" objname="testvolume" sourcetype="ontap:perf"

You can also create the following subsearch:

index=ontap sourcetype="ontap:perf" [| savedsearch "NFS Volumes Used by VMware (requires VMware app)" | search "Datastore name"="ISO" | rename Filer AS host, Volume as objname | fields host, objname]

Running this query gives you the performance information for the datastore known in your VMware environment as "ISO".