Configure data collection

Now that you have the Splunk App for NetApp Data ONTAP installed, the next step is to configure it to collect data from your ONTAP servers. Once configured you can validate that you are collecting the full set of data you specified and that the data being collected is correct.

In this topic you learn how to collect API data from your ONTAP servers. For information about forwarding syslog to the Splunk App for NetApp Data ONTAP, read Configure ONTAP log collection in this manual.

Collect data from your ONTAP environment

In Splunk Web on the search head, from the app menu, select NetApp. This brings you to the Splunk App for NetApp Data ONTAP. In the Settings menu click ONTAP Collection Configuration to navigate to that dashboard.

Add a data collection node

Data collection nodes (DCN) are managed by the scheduler. The scheduling node that runs the scheduler, typically on the search head, communicates with all data collection nodes over port 8008 (default setting). In your environment, if port 8008 is used by another service, you can configure another port for communication between the data collection node and the gateway. All data collection nodes do not have to communicate on the same port. You can configure the ports in the default stanza to implement the port change for all data collection nodes, or you can set the ports on a per stanza basis to configure the port for each data collection node individually.

Note: In a search head clustering (SHC) deployment, the DCN scheduler must be deployed on its own, dedicated search head. do not deploy the DCN Scheduler on any individual search heads within the SHC.

To set the port for the Hydra gateway, edit the configuration settings for the port on the scheduling node (usually implemented on the search head) in $SPLUNK_HOME/etc/apps/Splunk_TA_ontap/local/hydra_node.conf. The following is an example of the default setting for the app.

[default]
gateway_port = 8008

The hydra gateway port, default value 8008, uses the SSL certs that Splunk Web uses. Splunk Enterprise generates these SSL certs by default, but you can override them in the web.conf file. The only information that travels machine-to-machine over the gateway port is hydra job assignment, configuration, and performance information. Hydra passes no credentials or session keys for the target environment though the gateway port. Credentials pass only through the storage/passwords endpoint on Splunkd on default port 8089.

You can add data collection nodes to the scheduler and configure worker processes on each during the installation of the app. Each time you access a node, the credentials for Splunk and the add-ons on that node are validated. Do this for each data collection node on an individual basis. See Create a data collection node in this manual for more information.

To add a node:

1. On the ONTAP Collection Configuration dashboard, in the Data collection node panel, select Add Data Collection Node to add a new data collection node to register it with your scheduler.
2. Configure this node to collect data from your environment using the AddData Collection Node dialog. The node is not configured with default settings.
3. Enter the management URI to the Splunk forwarder on the data collection node. Give the full management URI of the Splunk installation - that is, the protocol (https is required), the address, and the management URI port number. For example, use

   https://testnode1:8089

   
Port 8089 is the default management port for a Splunk forwarder (not a universal forwarder).
4. Enter the Splunk Forwarder Username. The default username is admin.
5. Enter the Splunk Forwarder Password. Use the password you set when you deployed the data collection node. The DCN's original default password, changeme, is not valid.
6. Define the number of worker processes you want on the node: This is the number of processes you can run on the data collection node to process the data and forward it to the indexer(s). You can run a maximum of 8 processes per node at the default configuration. For more information see Splunk data collection node resource requirements.
7. Click Save to add a new node and display it on the dashboard.
8. Validate that the node is configured correctly: The details for the data collection node you configured are listed in the Data Collection Nodes panel. Look at the "Credential Validation" status for the node to see if you configured it correctly - the status will show "valid".

Troubleshooting

To troubleshoot your environment, you can set the field worker_log_level in hydra_node.conf to reflect a new log level for a data collection node. The default log level for a data collection node is INFO. DEBUG will be the most verbose logging level.

1. On the search head that administers the Distributed Collection Scheduler, create a local version of hydra_node.conf
2. Edit $SPLUNK_HOME/etc/apps/local/hydra_node.conf to set the log level of for all data collection nodes as per the following example:

[default]

gateway_port = 8008

capabilities = * 

log_level = DEBUG

Add ONTAP server(s)

Add one or more NetApp filers and configure them as the source of the data coming into Splunk. Note: When you add or remove a filer from your environment you must stop and restart the scheduler.

To add a Filer:

1. On the Collection Configuration dashboard, in the ONTAP Collection Configuration panel, select Add ONTAP Collection to add a new NetApp filer.
2. Configure filer settings in the Add ONTAP Collection dialog.
   1. Enter the fully qualified domain name for the ONTAP server(s), for example, test-na100.example.com. This can be a comma delimited list of ONTAP servers.
   2. Use Realm based Credentials. Check this box if the credentials you are using to access your ONTAP server(s) are part of a realm (controlled by an authentication policy set for a defined set of users).
   3. Enter the Realm: Enter the name that defines the realm.
   4. Enter an ONTAP Username: Enter the username for the user of the app. For example, enter "administrator" as a local user, or enter "splunkadmin@splunk.local" as an Active Directory domain user.
   5. Enter an ONTAP password. This is the password you use to access the the filer(s) in your NetApp environment.
   6. Collect All Performance Categories. This is enabled by default. All performance data is collected by the app unless you specifically request to only collect a subset of the data. To select specific performance data types, uncheck the box to disable automatic collection of all performance data. You can now select the performance categories you want to collect. Check the box beside each category of performance data (Volume, Disk, LUN, Aggregate, Vfiler, Qtree, Quota, System) you want to collect.
   7. Click Save to add a new ONTAP server and display it on the ONTAP Collection Configuration dashboard.
3. Validate that the ONTAP server is configured correctly. To validate that the ONTAP server is configured correctly the search head must be able to establish a connection to the filer. Validation can fail if the search head cannot directly connect to the filer. During the data collection process, all data is transferred through connections established by the data collection node ( on the firewall ports 8089 and 8008). You configured the server correctly if it displays in the list of ONTAP servers in the ONTAP Collection Configuration panel and credential validation is valid and and connection validation is valid.

Start the Scheduler

When your ONTAP servers are set up as data collection points and your data collection nodes are set up to gather API data, click Start Scheduler. The scheduler starts the data collection process and data is collected from the resources in your environment.

Note that if you add or modify credentials, restart the scheduler. This pushes the credentials out to the data collection nodes.

You have now completed all of the data collection configuration steps. Splunk is collecting ONTAP data from your environment and you can see the data when you look on the dashboards of the Splunk App for NetApp Data ONTAP.

You can read more about the scheduler in the introduction in this manual.

Manage your ONTAP environment

Check authentication and validation status of your ONTAP filers

In the Splunk App for NetApp Data ONTAP connectivity and credential validation are checked for a filer, or for a set of filers, during the configuration of the app, only after you click Save in the Edit ONTAP Collection dialog or the Add ONTAP Collection dialog. If there is a change to your environment after you have installed the app and configured data collection, for example, if the password on a filer changes, that change is not automatically reflected in the Collection Configuration page. This change can prevent you from logging in to a filer and prevent data collection.

If you notice that you have stopped receiving data from a filer, go to the Collection Configuration page and click Refresh Validation to check if there are connectivity issues that the refresh action can identify for that filer.

After fixing the connectivity issues you can wait up to 30 minutes for the data to come in to the app as when a data collection node recognizes that it can not connect to the filers, the worker processes limit that rate of attempts to login to the filer to once in a 30 minute period per filer.

You can also click Stop/Start scheduler to stop and restart data collection. This restarts data collection immediately from your ONTAP environment into Splunk.

Remember that if you add a new ONTAP filer as a target when the scheduler is running, you must stop and restart the scheduler to include the new filer as a data collection target. Refresh Validation does not update your Splunk environment.

Edit data collection node settings

To edit the configuration of your data collection node:

1. On the Collection Configuration dashboard, in the Data Collection Nodes panel, click on the node you want to modify.
2. You can edit the properties for the node, then click Save to update the configuration.

Delete a data collection node

Deleting a data collection node unregisters it from your scheduler. It no longer processes data or forwards data to your Splunk indexer.

To remove a data collection node from the scheduler configuration:

1. On the Collection Configuration dashboard, in the Data Collection Nodes panel, click on the node in the list, then click Delete node.
2. Confirm that you want to delete the node.
3. The node is removed from the list of nodes in the dashboard.

Edit ONTAP server settings

To edit filer settings:

1. On the Collection Configuration dashboard, in the ONTAP collection configuration panel, click on the filer you want to modify.
2. You can edit the properties for the filer, then click Save to update the configuration.

Delete a server

Deleting a server (filer) removes it from your Splunk environment. You will no longer collect data from this machine. When you add or remove a filer from your environment you must stop and restart the scheduler.

To delete a server:

1. On the Collection Configuration dashboard, in the ONTAP Collection Configuration panel, select the server from the list of target machines. The Edit ONTAP Collection dialog is displayed.
2. Click Delete Server.
3. Confirm that you want to delete the filer, then click Save.
4. The filer is removed as a data source and it is removed from the list of target machines in the dashboard.

Configure ONTAP servers

You can change the configuration settings of the ONTAP servers ( 7-mode or cluster mode) that you set up to work with the Splunk App for NetApp Data ONTAP. To do this, select the ONTAP server from the list of targets in the ONTAP Collection Configuration panel. The Edit ONTAP Collection dialog is displayed.

Configure performance category options

You can change the performance data collection options for your servers:

1. On the ONTAP Collection Configuration dashboard, in the ONTAP Collection Configuration panel, select the filer you want to reconfigure from the list of available filers. You can also use the ONTAP Server search box to find the specific filer.
2. To collect all performance data from the host, click "Collect All performance Categories".
3. To reduce the set of categories from which you want to collect performance data, uncheck the box. A list of performance categories is displayed (Volume, Disk, LUN, Aggregate, Vfiler, Qtree, Quota, System). Select the the check boxes that define the data set you want to collect.
4. Click Save to return to the dashboard.
5. To validate that you are only collecting data from the performance categories you selected, from the app menu, select Settings, then App Data Volume. The App Data Volume dashboard displays a list of data types and source types for the data you are collecting over the last 24 hour period.

Configure realm based access

You can set a server or group of servers to have realm based credentials. The ONTAP Credential Configuration page lists all of the realms defined within the app context.

1. In the Edit ONTAP Collection dialog, click Use Realm based Credentials.
2. Enter a realm.
3. Click Save.