Configure inputs

Configure receivers for ONTAP data

After installation, set up receiving on each of your indexers. Receivers, by convention, listen on port 9997, but any unused port is permitted. For more information see Set up receiving in the Splunk Forwarding data manual.

Configure Splunk to receive syslog data

Use a data collection node as the collection point for syslog data as it has Splunk_TA_ontap installed and the data input is set up. When you have installed the Splunk Add-on for NetApp Data ONTAP on the selected data collection node, enable the ontap:syslog data input by performing the below steps:

1. Copy the below stanza from $SPLUNK_HOME/etc/apps/Splunk_TA_ontap/default/inputs.conf to $SPLUNK_HOME/etc/apps/Splunk_TA_ontap/local/inputs.conf:

   #[udp://514]
   #index = ontap
   #sourcetype = ontap:syslog
   #connection_host = dns
   #disabled = 0
   

2. Uncomment this stanza in the local version of inputs.conf.

In very large environments, if you see a degradation in performance of your data collection node you can manually split the collection of your syslog data across multiple data collection nodes.

You can also use a dedicated forwarder or use the indexer that is connected to the data collection node as the collection point. In all cases, follow standard Splunk practices to configure Splunk to receive syslog data. Check that:

• Splunk is listening on UDP port 514.
• The sourcetype is set to ontap:syslog in the inputs.conf file.
• Splunk_TA_ontap is installed on the machine receiving syslog.

If you currently collect syslog data from the NetApp filers using a Splunk forwarder, you can continue to use the setup you have in your environment. Check that the forwarder receiving syslog is configured to send the data to the same indexers as the data collection node.

System log (syslog) management is important for troubleshooting performance problems across your network. Configure system log forwarding from NetApp to Splunk separately for your 7-mode and cluster mode filers. Log forwarding is done on the command line in your NetApp environment to forward to a Splunk forwarder. The forwarder must have network access to the storage device and be configured to listen on UDP port 514. Read the topic "Get data from TCP and UDP ports" in the Getting Data In manual for more information.

Turn on logging on data collection nodes

Turning on logging on the data collection node when you create the node assists in troubleshooting data collection issues. The collected data counts against your Splunk license.

1. Navigate to your data collection node.
2. Navigate to the SA-Hydra directory, and create a local directory.
3. Copy the outputs.conf file from SA-Hydra/default/ move it to SA-Hydra/local/.
4. Edit the SA-Hydra/local/outputs.conf file to uncomment the following lines:

   [tcpout]

   forwardedindex.3.whitelist = _internal

Configure timezones in syslog data

Ensure that the clock and timezone settings for your Splunk platform environment and your ONTAP servers agree so as to ensure accurate timestamping. In your Splunk platform, time offsets can cause indexing issues with defined data types. This is specifically true in the Splunk App for NetApp Data ONTAP for performance searches that use report acceleration. If the timezone information is not set correctly, your Splunk platform may incorrectly apply a timestamp and potentially exclude events from indexing. A light forwarder (LF) or universal forwarder (UF) do not parse events to get a timestamp. As a NetApp administrator, use NTP on your filers to check that the timezone settings on your ONTAP servers match the timezone information on your Splunk indexer(s).

Configure your NetApp environment to send syslog data to Splunk

In both 7-mode and in cluster mode, syslog is forwarded from your NetApp storage systems to Splunk by default on UDP port 514.

Configure syslog on 7-mode filers

1. Log in to the NetApp filer with the correct permissions.
2. To configure forwarding, on the command line enter the following, where forwarder is the IP address or DNS name of the receiving host:

   wrfile -a /etc/syslog.conf *.* @<forwarder>

Configure syslog forwarding on Cluster mode

In cluster mode there are many types of events, one of which is a syslog event. You can use specific Data ONTAP commands in the event family for managing these events. See the complete list of "Commands for managing events" in the NetApp online support documentation.

Configuring syslog in cluster mode is a two step process. First create a destination to where you will send the event. Once this is done you can forward the syslog event. You can forward to multiple forwarders, but you must specify a name for each one.

ONTAP Cluster Mode 9.0 and above

1. Log in to the NetApp filer with the correct permissions.
2. On the command line, set up the destination for the event, where <machine_name> is the IP address or DNS name of the receiving host:

   event notification destination create -name <name> -syslog <Forwarder_IP>

3. Filter forwarded data. You can forward all of the data from the cluster or you can forward a select set of data.

   1. Create filter event filter create -filter-name <filter_name>
   2. Forward syslogs using filters event notification create -filter-name <filter_name> -destinations <name_of_destination>
   3. View event destinations event notification destination show

4. Add a rule to the created filter in order to forward events.

   event filter rule add -filter-name <filtername> -type include -position <rule position> -severity <severity type>

ONTAP Cluster Mode 8.x and below

1. Log in to the NetApp filer with the correct permissions.
2. On the command line, set up the destination for the event as follows, where <machine_name> is the IP address or DNS name of the receiving host:

   event destination create -name int_fwd -syslog <machine_name>

3. Specify exactly what you want to forward. You can forward all of the data from the cluster or you can forward a select set of data. In this command you add the destination(s) established in the previous step to the event route. In this example we forward all of the data.
4. Filter the data you want to forward, and forward the data using this command:

   event route add-destinations -destinations int_fwd -messagename all

See the NetApp documentation, on "Managing event messages" for more detailed information.

Configure data collection intervals

The Splunk Add-on for NetApp Data ONTAP collects metrics for performance data and inventory data. Collection intervals are set in the $SPLUNK_HOME/etc/apps/Splunk_TA_ontap/default/ta_ontap_collection.conf file. The following two keys, megaperf_interval and megainv_interval, included in the default stanza of this file are used to set the intervals for data collection. The value that you assign to the interval reflects how you want to collect data in your environment. Use the number of objects (such as disk, volume, lun, qtree, or any other "inventory" object) on the filer to determine the interval settings for megaperf_interval and the megainv_interval for each filer. The most critical interval is the megaperf_interval. This interval governs the granularity of the performance data you collect and the total number of events coming into the system.

To collect data at the most granular level you must know the minimum performance interval. The minimum performance interval for performance data for this add-on is on average 0.1 to 0.2 seconds per object on the filer, given that no network issues impact collection.

For example, to calculate the recommended megaperf_interval in seconds for 3000 volumes, multiply the number of volumes by the minimum performance interval (0.2). This gives an interval value of 600 seconds.

The volume of data we collect for the megainv_interval is less than that of the megaperf_interval and the data is less frequent. We recommend a collection interval value that is 5 to 20 times the performance interval, so long as the interval is set to a value less than 60 minutes. To schedule performance and inventory collection on the same intervals, use the guidelines described above for collecting the data. Aggressive collection of inventory data (on the same frequency with which performance data is being collected) is not recommended.

The following is an example of the interval settings in the ta_ontap_collection.conf file

[default]

megaperf_interval = 60

megaperf_expiration = 55

megainv_interval = 600

megainv_expiration = 595

Distribute API requests across multiple data collection nodes

Distribute API requests across multiple data collection nodes (DCNs) to improve collection processing speed and to reduce collection fails. See the below example to use 2 DCNs to distribute performance and inventory collection.

Use two data collection nodes and distribute perf and inventory collection

1. Create two data collection nodes.
2. Navigate to the first DCN, and connect to NetApp Filer 1
3. Navigate to the default and copy the inputs.conf file.
4. Create a local directory and move the copied inputs.conf file to the local directory.
5. In the local/inputs.conf file, change the [default] stanza tasks from megainv, megaperf to megaperf.
   Example

    
   [default]
   tasks = megaperf
   

6. Navigate to the second DCN, and connect to NetApp Filer 1
7. Navigate to the default and copy the inputs.conf file.
8. Create a local directory and move the copied inputs.conf file to the local directory.
9. In the local/inputs.conf file, change the [default] stanza tasks from megainv, megaperf to megainv.
   Example

    
   [default]
   tasks = megainv
   

How to limit performance data collection

To reduce the volume of ONTAP performance data coming into the add-on, or to reduce the number of tasks that expire before they can complete, you can change the interval setting in the $SPLUNK_HOME/etc/apps/Splunk_TA_ontap/default/ta_ontap_collection.conf file. This affects the frequency with which ONTAP performance data is collected. Changing the collection interval also has an effect on the granularity of the ONTAP data you collect.

To change interval settings, create a local version of the file, if it does not already exist, ($SPLUNK_HOME/etc/apps/Splunk_TA_ontap/local/ta_ontap_collection.conf) and specify the changes here.

On the search head that runs the scheduler:

1. Edit the file $SPLUNK_HOME/etc/apps/Splunk_TA_ontap/local/ta_ontap_collection.conf. If it does not exist, create a new file.
2. Include the following stanza that affects ONTAP performance data collection:

   [default]

   megaperf_interval = 60

   megaperf_expiration = 55

   megainv_interval = 600

   megainv_expiration = 595

3. Restart Splunk.

Keep the value that you assign to expiration 5 to 10 seconds lower than the value you assign to the interval.