props.conf

props.conf

props.conf controls what parameters apply to events during indexing based on settings tied to each event's source, host, or sourcetype.

Note: You can only use wildcards for host or source. Use ... for paths and * for files.

props.conf.example

# Copyright (C) 2005-2008 Splunk Inc.  All Rights Reserved.  Version 3.0 
#
# The following are example props.conf configurations. Configure properties for your data.
#
# To use one or more of these configurations, copy the configuration block into
# props.conf in $SPLUNK_HOME/etc/system/local/. You must restart Splunk to enable configurations.
#
# To learn more about configuration files (including precedence) please see the documentation 
# located at http://www.splunk.com/base/Documentation/latest/Admin/HowDoConfigurationFilesWork.
########
# Line merging settings
########
# The following example linemerges source data into multi-line events for apache_error sourcetype.
[apache_error]
SHOULD_LINEMERGE = True
########
# Settings for tuning
########
# The following example limits the amount of characters indexed per event from host::small_events.
[host::small_events]
TRUNCATE = 256
# The following example turns off DATETIME_CONFIG (which can speed up indexing) from any path
# that ends in /mylogs/*.log.
[source::.../mylogs/*.log]
DATETIME_CONFIG = NONE
  
########
# Timestamp extraction configuration
########
# The following example sets Eastern Time Zone if host matches nyc*.
[host::nyc*]
TZ = US/Eastern
# The following example uses a custom datetime.xml that has been created and placed in a custom app
# directory. This sets all events coming in from hosts starting with dharma to use this custom file.
[host::dharma*]
DATETIME_CONFIG = <etc/apps/custom_time/datetime.xml>
########
# Transform configuration
########
# The following example creates a search field for host::foo if tied to a stanza in transforms.conf.
[host::foo]
TRANSFORMS-foo=foobar
# The following example creates an extracted field for sourcetype access_combined
# if tied to a stanza in transforms.conf.
[eventtype::my_custom_eventtype]
REPORT-baz = foobaz
########
# Sourcetype configuration
########
# The following example sets a sourcetype for the file web_access.log.
[source::.../web_access.log]
sourcetype = splunk_web_access 
# The following example untars syslog events.
[syslog]
invalid_cause = archive
unarchive_cmd = gzip -cd -
        
# The following example learns a custom sourcetype and limits the range between different examples
# with a smaller than default maxDist.
[custom_sourcetype]
LEARN_MODEL = true
maxDist = 30
# rule:: and delayedrule:: configuration
# The following examples create sourectype rules for custom sourcetypes with regex.
[rule::bar_some]
sourcetype = source_with_lots_of_bars
MORE_THAN_80 = ----
[delayedrule::baz_some]
sourcetype = my_sourcetype
LESS_THAN_70 = ####
########        
# File configuration
########
# Binary file configuration
# The following example eats binary files from any file that matches source::.../mylogs/*.log.
[source::.../mylogs/*.log]
NO_BINARY_CHECK = true 
    
# File checksum configuration
# The following example checks the entirety of every file in the web_access dir rather than 
# skipping files that appear to be the same.
[source::.../web_access/*]
CHECK_METHOD = entire_md5

props.conf.spec

# Copyright (C) 2005-2009 Splunk Inc.  All Rights Reserved.  Version 3.0 
#
# This file contains possible attribute/value pairs for configuring Splunk's processing properties
# via props.conf.
#
# There is a props.conf in $SPLUNK_HOME/etc/system/default/.  To set custom configurations, 
# place a props.conf in $SPLUNK_HOME/etc/system/local/. For help, see
# props.conf.example. 
# You can enable configurations changes made to props.conf by typing the following search string
# in Splunk Web:
#
# | extract reload=T 
#
# To learn more about configuration files (including precedence) please see the documentation 
# located at http://www.splunk.com/base/Documentation/latest/Admin/HowDoConfigurationFilesWork.

[<spec>]
* This stanza enables properties for a given <spec>. 
* A props.conf file can contain multiple stanzas for any number of different <spec>.
* Follow this stanza name with any number of the following attribute/value pairs.
* If you do not set an attribute for a given <spec>, the default is used.

<spec> can be:
1. <sourcetype>, the sourcetype of an event.
2. host::<host>, where <host> is the host for an event.
3. source::<source>, where <source> is the source for an event.
4. rule::<rulename>, where <rulename> is a unique name of a sourcetype classification rule.
5. delayedrule::<rulename>, where <rulename> is a unique name of a delayed sourcetype classification rule.  
These are only considered as a last resort before generating a new sourcetype based on the source seen.

Precedence:

For settings that are specified in multiple categories of matching stanzas,
host:: spec settings override sourcetype:: spec settings.
Additionally, source:: spec settings override both host:: and
sourcetype:: settings.

For a given event, only one [sourcetype], one [source::...] and one [host::...]
stanza will apply.  When multiple stanzas might match, which is applied is not
defined.
   

NOTE: When setting a <spec> (EXCEPT sourcetype), you can use the following regex-type syntax:

... = recurses through directories until the match is met.

* = matches anything but / 0 or more times.

| = or 

( ) = used to limit scope of |.

Example: [source::....(?<!tar.)(gz|tgz)] 

Match language:

These match expressions must match the entire key value, not just a substring.

For those familiar with regular expressions, these are a full implementation
(PCRE) with the translation of ..., * and .
Thus . matches a period, * non-directory seperators, and ... any number of any characters.
For more information see the wildcards section at: 

http://www.splunk.com/base/Documentation/latest/Admin/FilesAndDirectories#Inputs.conf

#******************************************************************************
# The possible attributes/value pairs for props.conf, and their default values, are:
#******************************************************************************

# International characters

CHARSET = <string>
* When set, Splunk assumes the input from the given <spec> is in the specified encoding.  
* A list of valid encodings can be retrieved using the command "iconv -l" on most *nix systems.  
* If an invalid encoding is specified, a warning is logged during initial configuration and further input from that <spec> is discarded.  
* If the source encoding is valid, but some characters from the <spec> are not valid in the specified encoding, then the characters are escaped as hex (e.g. "\xF3").
* When set to "AUTO", Splunk attempts to automatically determine the character encoding and convert text from that encoding to UTF-8.  
* For a complete list of the character sets Splunk automatically detects, see the online documentation.
* Defaults to AUTO on Windows and UTF-8 on non-Windows OSes.

#******************************************************************************
# Line breaking
#******************************************************************************

# Use the following attributes to define the length of events.

TRUNCATE = <non-negative integer>
* Change the default maximum line length.  
* Set to 0 if you do not want truncation ever (very long lines are, however, often a sign of garbage data).
* Defaults to 10000.

LINE_BREAKER = <regular expression>
* If not set, data is broken into an event for each line, delimited by \r or \n.
* The contents of the first matching group does not occur in either the previous or next events.
* NOTE: There is a significant speed boost by using the LINE_BREAKER to delimit multiline events rather than using line merging to reassemble individual lines into events.
* If set, the given regex is used to break the raw stream into events.
* Wherever the regex matches, the start of the first match is considered the start of the next event.
* The regex must contain a matching group. 
* Defaults to ([\r\n]+).

LINE_BREAKER_LOOKBEHIND = <integer>
* Change the default lookbehind for the regex based linebreaker. 
* When there is leftover data from a previous raw chunk, this is how far before the end the raw chunk (with the next chunk concatenated) we should begin applying the regex.
* Defaults to 100.

# Use the following attribute to define multi-line events with additional attributes and values.

SHOULD_LINEMERGE = true | false
* When set to true, Splunk combines several lines of data into a single event, based on the following configuration attributes.
* Defaults to true.
  	
# When SHOULD_LINEMERGE = True, use the following attributes to define the multi-line events.

AUTO_LINEMERGE = true | false
* Directs Splunk to use automatic learning methods to determine where to break lines in events.
* Defaults to true.

BREAK_ONLY_BEFORE_DATE = true | false
* When set to true, Splunk creates a new event if and only if it encounters a new line with a date.
* Defaults to false.

BREAK_ONLY_BEFORE = <regular expression>
* When set, Splunk creates a new event if and only if it encounters a new line that matches the regular expression.
* Defaults to empty.

MUST_BREAK_AFTER = <regular expression>
* When set, and the regular expression matches the current line, Splunk creates a new event for the next input line.
* Splunk may still break before the current line if another rule matches.
* Defaults to empty.

MUST_NOT_BREAK_AFTER = <regular expression>
* When set and the current line matches the regular expression, Splunk does not break on any subsequent lines until the MUST_BREAK_AFTER expression matches.
* Defaults to empty.

MUST_NOT_BREAK_BEFORE = <regular expression>
* When set and the current line matches the regular expression, Splunk does not break the last event before the current line.
* Defaults to empty.

MAX_EVENTS = <integer>
* Specifies the maximum number of input lines to add to any event. 
* Splunk breaks after the specified number of lines are read.
* Defaults to 256.
  	

#******************************************************************************
# Timestamp extraction configuration
#******************************************************************************

DATETIME_CONFIG = <filename relative to $SPLUNK_HOME>
* Specifies which file configures the timestamp extractor.
* This configuration may also be set to "NONE" to prevent the timestamp extractor from running or "CURRENT" to assign the current system time to each event.
* Defaults to /etc/datetime.xml (eg $SPLUNK_HOME/etc/datetime.xml).

MAX_TIMESTAMP_LOOKAHEAD = <integer>
* Specifies how far (in characters) into an event Splunk should look for a timestamp.
* Defaults to 150.

TIME_PREFIX = <regular expression>
* Specifies the necessary condition for timestamp extraction. 
* The timestamping algorithm only looks for a timestamp after the first regex match.
* Defaults to empty.

TIME_FORMAT = <strptime-style format>
* Specifies a strptime format string to extract the date. 
* This method of date extraction does not support in-event timezones. 
* TIME_FORMAT starts reading after the TIME_PREFIX. 
* The <strptime-style format> must contain the hour, minute, month, and day.
* Defaults to empty.
* NOTE: If you use TIME_FORMAT Splunk assumes your strptime is correctly formatted, and all MAX* settings (below) are ignored.  

TZ = <timezone identifier>
* The algorithm for determining the time zone for a particular event is as follows:
* If the event has a timezone in its raw text (e.g., UTC, -08:00), use that.
* If TZ is set to a valid timezone string, use that.
* Otherwise, use the timezone of the system that is running splunkd.
* Defaults to empty.

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days past, from the current date, for an extracted date to be valid.  
* If set to 10, for example, Splunk ignores dates that are older than 10 days ago.
* Defaults to 2000.
* IMPORTANT: If your data is older than 2000 days, change this setting.

MAX_DAYS_HENCE = <integer>
* Specifies the maximum number of days in the future, from the current date, for an extracted date to be valid.  
* If set to 3, for example, dates that are more than 3 days in the future are ignored.
* False positives are less likely with a tighter window.
* The default value includes dates from one day in the future.  
* If your servers have the wrong date set or are in a timezone that is one day ahead, increase this value to at least 3.
* Defaults to 2.

MAX_DIFF_SECS_AGO = <integer>
* If the event's timestamp is more than <integer> seconds BEFORE the previous timestamp, only accept it if it has the same exact time format as the majority of timestamps from the source.
* IMPORTANT: If your timestamps are wildly out of order, consider increasing this value.
* Defaults to 3600 (one hour).

MAX_DIFF_SECS_HENCE = <integer>
* If the event's timestamp is more than <integer> seconds AFTER the previous timestamp only accept it if it has the same exact time format as the majority of timestamps from the source.
* IMPORTANT: If your timestamps are wildly out of order, or you have logs that are written less than once a week, consider increasing this value.
* Defaults to 604800 (one week).


#******************************************************************************
# Transform configuration
#******************************************************************************

# Use the TRANSFORMS class to create indexed fields.  Use the REPORT class to create extracted fields.
# Please note that extracted fields are recommended as best practice.
# Note: Indexed fields have performance implications and are only recommended in specific circumstances.
# You may want to use indexed fields if you search for expressions like foo!="bar" or NOT foo="bar" and the field foo nearly always takes on the value bar. 
# Another common reason to use indexed fields is if the value of the field exists outside of the field more often than not. 
# For example, if you commonly search for foo="1", but 1 occurs in many events that do not have foo="1", you may want to index foo. 
# For more information, see documentation at: http://www.splunk.com/doc/latest/admin/ExtractFields
# For examples, see props.conf.spec and transforms.conf.spec.

Precedence rules for classes:

* For each class, Splunk takes the configuration from the highest precedence configuration block (see precedence rules at the beginning of this file).
* If a particular class is specified for a source and a sourcetype, the class for source wins out. 
* Similarly, if a particular class is specified in ../local/ for a <spec>, it overrides that class  in ../default/.


TRANSFORMS-<value> = <unique_stanza_name>
*  <unique_stanza_name> is the name of your stanza from transforms.conf.
* <value> is any value you want to give to your stanza to identify its name-space.
* Transforms are applied in the specified order.
* If you need to change the order, control it by rearranging the list.
  
REPORT-<value> = <unique_stanza_name>
*  <unique_stanza_name> is the name of your stanza from transforms.conf.
* <value> is any value you want to give to your stanza to identify its name-space.
* Transforms are applied in the specified order.
* If you need to change the order, control it by rearranging the list.

KV_MODE = none | auto | multi
* Specifies the key/value extraction mode for the data. 
* Set KV_MODE to one of the following:
 * none: if you want no key/value extraction to take place.
 * auto: extracts key/value pairs separated by equal signs.
 * multi: invokes multikv to expand a tabular event into multiple events.
* Defaults to auto.
  	
  	
CHECK_FOR_HEADER = true | false
* Set to true to enable header-based field extraction for a file.
* If the file has a list of columns and each event contains a field value (without field name), Splunk picks a suitable header line to use to for extracting field names.
* Defaults to false.


#******************************************************************************
# Segmentation configuration
#******************************************************************************

SEGMENTATION = <string>
* Specifies the segmenter from segmenters.conf to use at index time.
* Set segmentation for any of the <spec> outlined at the top of this file.

SEGMENTATION-<segment selection> = <string>
* Specifies that Splunk Web should use the a specific segmenter (from segmenters.conf) for the given <segment selection> choice. 
* Default <segment selection> choices are: all, inner, outer, none.
  	
  	
#******************************************************************************
# Binary file configuration
#******************************************************************************

NO_BINARY_CHECK = true | false
* Can only be set for a [<source>::...] stanza.
* When set to true, Splunk processes binary files.
* By default, binary files are ignored.
* Defaults to false.
    
    
#******************************************************************************
# File checksum configuration
#******************************************************************************

CHECK_METHOD = entire_md5 | modtime
* By default, if the checksums of the first and last 256 bytes of a file match existing stored checksums, Splunk lists the file as already indexed and thus ignores it.
* Set this to "entire_md5" to use the checksum of the entire file.
* Alternatively, set this to "modtime" to check only the modification time of the file.
* Defaults to endpoint_md5.
    
#******************************************************************************
# Sourcetype configuration
#******************************************************************************

sourcetype = <string>
* Can only be set for a [<source>::...] stanza.
* Anything from that <source> is assigned the specified sourcetype.
* Defaults to empty.
    
# The following attribute/value pairs can only be set for a stanza that begins with [<sourcetype>]:

invalid_cause = <string>
* Can only be set for a [<sourcetype>] stanza.
* Splunk does not index any data with invalid_cause set.
* Set <string> to "archive" to send the file to the archive processor (specified in unarchive_cmd).
* Set to any other string to throw an error in the splunkd.log if running Splunklogger in debug mode.
* Defaults to empty.
  	
is_valid = true | false
* Automatically set by invalid_cause.
* DO NOT SET THIS.
* Defaults to true.

unarchive_cmd = <string>
* Only called if invalid_cause is set to "archive".
* <string> specifies the shell command to run to extract an archived source.
* Must be a shell command that takes input on stdin and produces output on stdout.
* DOES NOT WORK ON BATCH PROCESSED FILES. Use preprocessing_script.
* Defaults to empty.

LEARN_MODEL = true | false
* For known sourcetypes, the fileclassifier adds a model file to the learned directory.
* To disable this behavior for diverse sourcetypes (such as sourcecode, where there is no good exemplar to make a sourcetype) set LEARN_MODEL = false.
* Defaults to empty.

maxDist = <integer>
* Determines how different a sourcetype model may be from the current file.  
* The larger the value, the more forgiving.
* For example, if the value is very small (e.g., 10), then files of the specified  sourcetype should not vary much.
* A larger value indicates that files of the given sourcetype vary quite a bit.
* Defaults to 300.


# rule:: and delayedrule:: configuration

MORE_THAN<optional_unique_value>_<number> = <regular expression> (empty)
LESS_THAN<optional_unique_value>_<number> = <regular expression> (empty)

An example:

[rule::bar_some]
sourcetype = source_with_lots_of_bars
# if more than 80% of lines have "----", but fewer than 70% have "####"
# declare this a "source_with_lots_of_bars"
MORE_THAN_80 = ----
LESS_THAN_70 = ####

A rule can have many MORE_THAN and LESS_THAN patterns, and all are required for the rule to match.


#******************************************************************************
# Internal settings
#******************************************************************************

# NOT YOURS.  DO NOT SET.

_actions = <string> ("new,edit,delete")
* Internal field used for user-interface control of objects.
* Defaults to "new,edit,delete".

pulldown_type = <bool>
* Internal field used for user-interface control of sourcetypes.
* Defaults to empty.

Note: See next page for continuation of props.conf.spec.

• Was this topic useful?
• Post a comment