Performance best practices

Splunk Analytics for Hadoop reaches End of Life on January 31, 2025.

When your raw HDFS data is subjected to the search process, the data passes through index-time processing. (Index time extractions run at search time and cannot be turned off.)

In order to more efficiently process this data, you should optimize your index-time settings, particularly timestamping and aggregation. The following settings added to your data source in props.conf can be configured to improve performance:

• DATETIME_CONFIG
• MAX_TIMESTAMP_LOOKAHEAD
• TIME_PREFIX
• TIME_FORMAT
• SHOULD_LINEMERGE
• ANNOTATE_PUNCT

For example, for single line, non-timestamped data, the following settings can improve throughput roughly four times over:

[source::MyDataSource]
ANNOTATE_PUNCT   = false
SHOULD_LINEMERGE = false
DATETIME_CONFIG  = NONE

Note: If you need to use timestamping, we strongly recommend that you use TIME_PREFIX and TIME_FORMAT to improve processing.

The table below shows examples of possible timestamping and breaking options and how long (in seconds) that combination can take when processing a file with 10 million single line events:

Timestamping and breaking options:	Time:
Default configuration	190 seconds
MAX_TIMESTAMP_LOOKAHEAD = 30	179
MAX_TIMESTAMP_LOOKAHEAD = 30 SHOULD_LINEMERGE = false	105
MAX_TIMESTAMP_LOOKAHEAD = 30 SHOULD_LINEMERGE = false TIME_PREFIX = ^	107
MAX_TIMESTAMP_LOOKAHEAD = 30 SHOULD_LINEMERGE = false TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z	51
MAX_TIMESTAMP_LOOKAHEAD = 30 SHOULD_LINEMERGE = false TIME_PREFIX = ^ TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z	53
MAX_TIMESTAMP_LOOKAHEAD = 30 SHOULD_LINEMERGE = false TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z ANNOTATE_PUNCT = false	44
SHOULD_LINEMERGE = false	109
SHOULD_LINEMERGE = false TIME_PREFIX = ^	99
SHOULD_LINEMERGE = false TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z	54
SHOULD_LINEMERGE = false TIME_PREFIX = ^ TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z	54
MAX_TIMESTAMP_LOOKAHEAD = 30 SHOULD_LINEMERGE = false DATETIME_CONFIG = NONE	49
SHOULD_LINEMERGE = false DATETIME_CONFIG = CURRENT	50
MAX_TIMESTAMP_LOOKAHEAD = 30 SHOULD_LINEMERGE = false DATETIME_CONFIG = NONE ANNOTATE_PUNCT = false	35

Disable streaming to speed up searches

If you want data only from MapReduce jobs, without previews, you can disable the Streaming feature of Splunk Analytics for Hadoop to speed up searches.

By default Splunk Analytics for Hadoop uses Mix Mode, which combining of Streaming (Splunk only) and Reporting (Hadoop MR jobs) modes. If you do not require a preview, you can disable the streaming part of Splunk Anaytics for Hadoop.

To enable or disable streaming:

• Mix Mode: vix.mode = report and vix.splunk.search.mixedmode = 1
• Report Mode only: vix.mode = report and vix.splunk.search.mixedmode = 0
• Streaming Mode only: vix.mode = stream