Getting Data In

Get data from TCP and UDP ports

You can enable Splunk to accept an input on any TCP or UDP port. Splunk consumes any data sent on these ports. Use this method for syslog (default port is UDP 514) or set up netcat and bind to a port.

TCP is the protocol underlying Splunk's data distribution and is the recommended method for sending data from any remote machine to your Splunk server. Splunk can index remote data from syslog-ng or any other application that transmits via TCP.

Splunk supports monitoring over UDP, but recommends using TCP instead whenever possible. UDP is generally undesirable as a transport because:

• It doesn't enforce delivery.
• It's not encrypted.
• There's no accounting for lost datagrams.

Refer to "Working with UDP connections" on the Splunk Community Wiki for recommendations if you must use UDP.

Add a network input using Splunk Web

To add inputs from network ports using Splunk Web:

A. Go to the Add New page

You add a network input from the Add New page in Splunk Web. You can get there through two routes:

• Splunk Manager
• Splunk Home

It doesn't matter which route you use to get there; the Add New page itself is the same either way.

Via Splunk Manager:

1. Click Manager in the upper right-hand corner of Splunk Web.

2. In the Data section of the Manager page, click Data Inputs.

3. Pick TCP or UDP.

4. Click the New button to add an input.

Via Splunk Home:

1. Click the Add Data link in Splunk Home. This brings you to a page called "Data recipes".

2. Click either the From a TCP port or the From a UDP port link to add an input.

B. Specify the network input

1. Enter a port number. The user you run Splunk as must have access to the port. On a Unix system, you must run as root to access a port under 1024.

2. If this is a TCP input, you can specify whether this port should accept connections from all hosts or one host. If you specify one host, enter the IP address of the host.

3. Enter a new Source name to override the default source value, if necessary.

Important: Consult Splunk support before changing this value.

4. To access other settings, check More settings. A number of additional settings appear. You can usually go with the defaults for these settings. If you want to set them explicitly, here's what they're for:

a. You can set the Host by selecting a radio button:

• IP. Sets the input processor to rewrite the host with the IP address of the remote server.

• DNS. Sets the host to the DNS entry of the remote server.

• Custom. Sets the host to a user-defined label.

b. You can set the Source type. Source type is a default field added to events. Source type is used to determine processing characteristics, such as timestamps and event boundaries. For information on overriding Splunk's automatic source typing, see "Override automatic source type assignment" in this manual.

c. You can set the Index. Leave the value as "default" unless you have defined multiple indexes to handle different types of events. In addition to indexes for user data, Splunk has a number of utility indexes, which also appear in this dropdown box.

5. Click Save.

Add a network input using the CLI

To access Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command.

If you get stuck, Splunk's CLI has built-in help. Access the main CLI help by typing splunk help. Individual commands have their own help pages as well; type splunk help <command>.

The following CLI commands are available for network input configuration:

Command	Command syntax	Action
add	add tcp | udp $SOURCE [-parameter value] ...	Add inputs from $SOURCE.
edit	edit tcp | udp $SOURCE [-parameter value] ...	Edit a previously added input for $SOURCE.
remove	remove tcp | udp $SOURCE	Remove a previously added data input.
list	list tcp | udp	List the currently configured monitor.

Change the configuration of each data input type by setting additional parameters. Parameters are set with this syntax:

  -<parameter> <value>

Parameter	Required?	Description
$SOURCE	Yes	Port number to listen for data to index. The user you run Splunk as must have access to this port. On a Unix system, you must run as root to access a port under 1024.
sourcetype	No	Specify a sourcetype field value for events from the input source.
index	No	Specify the destination index for events from the input source.
hostname	No	Specify a host name to set as the host field value for events from the input source.
remotehost	No	Specify an IP address to exclusively accept data from.
resolvehost	No	Set True or False (T | F). Default is False. Set True to use DNS to set the host field value for events from the input source.

Examples

• Configure a UDP input to watch port 514 and set the source type to "syslog":

     ./splunk add udp 514 -sourcetype syslog

• Set the UDP input's host value via DNS. Use auth with your username and password:

     ./splunk edit udp 514 -resolvehost true -auth admin:changeme

Check the Best Practices Wiki for information about the best practices for using UDP when configuring Syslog input.

Add a network input using inputs.conf

To add an input, add a stanza for it to inputs.conf in $SPLUNK_HOME/etc/system/local/, or your own custom application directory in $SPLUNK_HOME/etc/apps/. If you have not worked with Splunk's configuration files before, read "About configuration files" in the Admin manual before you begin.

You can set any number of attributes and values following an input type. If you do not specify a value for one or more attributes, Splunk uses the defaults that are preset in $SPLUNK_HOME/etc/system/default/ (noted below).

TCP

[tcp://<remote server>:<port>]
<attrbute1> = <val1>
<attrbute2> = <val2>
...

This type of input stanza tells Splunk to listen to <remote server> on <port>. If <remote server> is blank, Splunk listens to all connections on the specified port.

Note: The user you run Splunk as must have access to the listening port. On a Unix system, you must run as root to access a port under 1024.

host = <string>

• Sets the host key/field to a static value for this stanza.
• Sets the host key's initial value. The key is used during parsing/indexing, in particular to set the host field. It is also the host field used at search time.
• The <string> is prepended with 'host::'.
• If not set explicitly, this defaults to the IP address or fully qualified domain name of the host where the data originated.

index = <string>

• Set the index where events from this input will be stored.
• The <string> is prepended with 'index::'.
• Defaults to main, or whatever you have set as your default index.
• For more information about the index field, see "How indexing works" in the Admin manual.

sourcetype = <string>

• Sets the sourcetype key/field for events from this input.
• Explicitly declares the source type for this data, as opposed to allowing it to be determined automatically. This is important both for searchability and for applying the relevant formatting for this type of data during parsing and indexing.
• Sets the sourcetype key's initial value. The key is used during parsing/indexing, in particular to set the source type field during indexing. It is also the source type field used at search time.
• The <string> is prepended with 'sourcetype::'.
• If not set explicitly, Splunk picks a source type based on various aspects of the data. There is no hard-coded default.
• For more information about source types, see "Why source types matter", in this manual.

source = <string>

• Sets the source key/field for events from this input.
• Note: Overriding the source key is generally not recommended. Typically, the input layer will provide a more accurate string to aid in problem analysis and investigation, accurately recording the file from which the data was retreived. Consider use of source types, tagging, and search wildcards before overriding this value.
• The <string> is prepended with 'source::'.
• Defaults to the input file path.

queue = [parsingQueue|indexQueue]

• Specifies where the input processor should deposit the events that it reads.
• Set to "parsingQueue" to apply props.conf and other parsing rules to your data.
• Set to "indexQueue" to send your data directly into the index.
• Defaults to parsingQueue.

connection_host = [ip|dns|none]

• "ip" sets the host to the IP address of the remote server.
• "dns" sets the host to the DNS entry of the remote server.
• "none" leaves the host as specified.
• Defaults to "ip".

TCP over SSL

[tcp-ssl:<port>]

Use this stanza type if you are receiving encrypted, unparsed data from a forwarder or third-party system. Set <port> to the port on which the forwarder or third-party system is sending unparsed, encrypted data.

UDP

[udp://<remote server>:<port>]
<attrbute1> = <val1>
<attrbute2> = <val2>
...

This type of input stanza is similar to the TCP type, except that it listens on a UDP port.

Note:

• If <remote server> is specified, the specified port will only accept data from that server.
• If <remote server> is empty - [udp://<port>] - the port will accept data sent from any server.

host = <string>

• Sets the host key/field to a static value for this stanza.
• Sets the host key's initial value. The key is used during parsing/indexing, in particular to set the host field. It is also the host field used at search time.
• The <string> is prepended with 'host::'.
• If not set explicitly, this defaults to the IP address or fully qualified domain name of the host where the data originated.

index = <string>

• Set the index where events from this input will be stored.
• The <string> is prepended with 'index::'.
• Defaults to main, or whatever you have set as your default index.
• For more information about the index field, see "How indexing works" in the Admin manual.

sourcetype = <string>

• Sets the sourcetype key/field for events from this input.
• Explicitly declares the source type for this data, as opposed to allowing it to be determined automatically. This is important both for searchability and for applying the relevant formatting for this type of data during parsing and indexing.
• Sets the sourcetype key's initial value. The key is used during parsing/indexing, in particular to set the source type field during indexing. It is also the source type field used at search time.
• The <string> is prepended with 'sourcetype::'.
• If not set explicitly, Splunk picks a source type based on various aspects of the data. There is no hard-coded default.
• For more information about source types, see "Why source types matter", in this manual.

source = <string>

• Sets the source key/field for events from this input.
• Note: Overriding the source key is generally not recommended. Typically, the input layer will provide a more accurate string to aid in problem analysis and investigation, accurately recording the file from which the data was retreived. Consider use of source types, tagging, and search wildcards before overriding this value.
• The <string> is prepended with 'source::'.
• Defaults to the input file path.

queue = [parsingQueue|indexQueue]

• Specifies where the input processor should deposit the events that it reads.
• Set to "parsingQueue" to apply props.conf and other parsing rules to your data.
• Set to "indexQueue" to send your data directly into the index.
• Defaults to parsingQueue.

_rcvbuf = <integer>

• Specify the receive buffer for the UDP port, measured in bytes.
• If the value is 0 or negative, it is ignored.
• Defaults to 1,572,864.
• Note: If the default value is too large for an OS, Splunk will try to set the value to 1572864/2. If that value also fails, Splunk will retry with 1572864/(2*2). It will continue to retry by halving the value until it succeeds.

no_priority_stripping = [true|false]

• Setting for receiving syslog data.
• If this attribute is set to true, Splunk does NOT strip the <priority> syslog field from received events.
• Depending on how this attribute is set, Splunk sets event timestamps differently:
  • When set to true, Splunk honors the timestamp as it comes from the source.
  • When set to false, Splunk assigns events the local time.
• Default is false (Splunk strips <priority>).

no_appending_timestamp = [true|false]

• If this attribute is set to true, Splunk does NOT append a timestamp and host to received events.
• Note: Do NOT include this attribute if you want to append timestamp and host to received events.
• Default is false.

UDP packets and line merging

When using UDP as an input, there is an implicit expectation that each UDP packet will be indexed as an independent event. However, this is not true. Splunk performs event merging on the datastream, so events will be merged together if they don't have a clear timestamp.

In the case of single-line events, you can avoid this problem by editing the underlying source type in props.conf and setting the SHOULD_LINEMERGE attribute to false. By doing so, you will keep Splunk from merging packets together.

Answers

Have questions? Visit Splunk Answers and see what and answers the Splunk community has about questions UDP inputs, TCP inputs, and inputs in general,