Add or edit an HDFS provider in Splunk Web

Splunk Analytics for Hadoop reaches End of Life on January 31, 2025.

You can set up multiple providers with multiple indexes for one provider. When you add a virtual index, have the following information at hand:

• The host name and port for the NameNode of the Hadoop cluster.
• The host name and port for the JobTracker of the Hadoop cluster.
• Installation directories of Hadoop command line libraries and Java installation.
• Path to a writable directory on the DataNode/TaskTracker *nix filesystem, the one for which the Hadoop user account has read and write permission.
• Path to a writable directory in HDFS that can be used exclusively by Splunk on this search head.

You can also add HDFS providers and virtual indexes by editing indexes.conf. See Set up a virtual index for instructions on setting up virtual indexes in the configuration file.

Add a provider

1. In the top menu, select Settings > Virtual Indexes.

2. Select the Providers tab in the Virtual Indexes page and click New Provider or the name of the provider you want to edit.

3. The Add New/Edit Provider page, give your provider a Name.

4. Select the Provider Family in the drop down list (note that this field cannot be edited).

5. Provide the following Environment Variables:

• Java Home: provide the path to your Java instance.

• Hadoop Home: Provide the path to your Hadoop client directory.

6. Provide the following Hadoop Cluster Information:

• Hadoop Version: Tell Splunk Analytics for Hadoop which version of Hadoop the cluster is running one of: Hadoop 1.0, Hadoop 2.0 with MRv1 or Hadoop 2.0 with Yarn.

• JobTracker: Provide the path to the Job Tracker.

• File System: Provide the path to the default file system.

7. Provide the following Splunk Settings:

• HDFS working directory: This is a path in HDFS (or whatever the default file system is) that you want to use as a working directory.

• Job queue: This is job queue where you want the MapReduce jobs for this provider to be submitted to.

8. Click Add Secure Cluster to configure security for the cluster and provide your Kerberos Server configuration.

9. The Additional Settings fields specify your provider configuration variables. Splunk Analytics for Hadoop populates these preset configuration variables for each provider you create. You can leave the preset variables in place or edit them as needed. If you want to learn more about these settings, see Provider Configuration Variables in the reference section of this manual.

Note: If you are configuring Splunk Analytics for Hadoop to work with YARN, you must add new settings. See Required configuration variables for YARN in this manual.

9. Click Save.