Splunk® Enterprise

Splunk Analytics for Hadoop

Download manual as PDF

Download topic as PDF

Working with Hive and Parquet data

Data Preprocessors

When Splunk Analytics for Hadoop initializes a search for non-HDFS input data, it uses the information contained in the FileSplitGenerator class to determine how to split data for parallel processing.

The default FileSplitGenerator contains the same data split logic defined in Hadoop's FileInputFormat This means that it works for any data format that can be read by Hadoop's InputFormat implementation (which has same split logic as FileInputFormat).

Since the default FileSplitGenerator does not work for Hive or Parquet files, Splunk Analytics for Hadoop provides HiveSplitGenerator and ParquetSplitGenerator for Hive and Parquet. Any custom Hive files with file-based split logic (such as files created with Hadoop FileOutputFormat and its subclasses) works with the HiveSplitGenerator. If you have custom Hive file formats that do not use file-based data split logic, you can implement a custom SplitGenerator that uses your split logic.

Parquet files created by all tools (including Hive) work with (and only with) ParquetSplitGenerator.

Last modified on 07 August, 2019
PREVIOUS
Configure Splunk Analytics for Hadoop to read Hadoop Archive (HAR) files
  NEXT
Configure Hive connectivity

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.5.6, 6.5.7, 6.5.8, 6.5.9, 6.5.10, 6.6.0, 6.6.1, 6.6.2, 6.6.3, 6.6.4, 6.6.5, 6.6.6, 6.6.7, 6.6.8, 6.6.9, 6.6.10, 6.6.11, 6.6.12, 7.0.0, 7.0.1, 7.0.2, 7.0.3, 7.0.4, 7.0.5, 7.0.6, 7.0.7, 7.0.8, 7.0.9, 7.0.10, 7.0.11, 7.0.13, 7.1.0, 7.1.1, 7.1.2, 7.1.3, 7.1.4, 7.1.5, 7.1.6, 7.1.7, 7.1.8, 7.1.9, 7.1.10, 7.2.0, 7.2.1, 7.2.2, 7.2.3, 7.2.4, 7.2.5, 7.2.6, 7.2.7, 7.2.8, 7.2.9, 7.2.10, 7.3.0, 7.3.1, 7.3.2, 7.3.3, 7.3.4, 7.3.5, 8.0.0, 8.0.1, 8.0.2, 8.0.3


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters