Hunk®(Legacy)

Hunk User Manual

Download manual as PDF

Download topic as PDF

Performance best practices

When your raw HDFS data is subjected to the search process, the data passes through index-time processing. (Index time extractions run at search time and cannot be turned off.)

In order to more efficiently process this data, you should optimize your index-time settings, particularly timestamping and aggregation. The following settings added to your data source in props.conf can be configured to improve performance:

  • DATETIME_CONFIG
  • MAX_TIMESTAMP_LOOKAHEAD
  • TIME_PREFIX
  • TIME_FORMAT
  • SHOULD_LINEMERGE
  • ANNOTATE_PUNCT

For example, for single line, non-timestamped data, the following settings can improve throughput roughly four times over:

[source::MyDataSource]
ANNOTATE_PUNCT   = false
SHOULD_LINEMERGE = false
DATETIME_CONFIG  = NONE

Note: If you need to use timestamping, we strongly recommend that you use TIME_PREFIX and TIME_FORMAT to improve processing.

The table below shows examples of possible timestamping and breaking options and how long (in seconds) that combination can take when processing a file with 10 million single line events:

Timestamping and breaking options: Time:

Default configuration

190 seconds

MAX_TIMESTAMP_LOOKAHEAD = 30

179

MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false

105

MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
TIME_PREFIX = ^

107

MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z

51

MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z

53

MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z
ANNOTATE_PUNCT = false

44

SHOULD_LINEMERGE = false

109

SHOULD_LINEMERGE = false
TIME_PREFIX = ^

99

SHOULD_LINEMERGE = false
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z

54

SHOULD_LINEMERGE = false
TIME_PREFIX = ^
TIME_FORMAT = %a, %d %b %Y %H:%M:%S %Z

54

MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
DATETIME_CONFIG = NONE

49

SHOULD_LINEMERGE = false
DATETIME_CONFIG = CURRENT

50

MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
DATETIME_CONFIG = NONE
ANNOTATE_PUNCT = false

35
PREVIOUS
Troubleshoot Hunk
  NEXT
Provider Configuration Variables

This documentation applies to the following versions of Hunk®(Legacy): 6.0, 6.0.1, 6.0.2, 6.0.3, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.2, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters