Set up streaming

A modular input can stream data to a Splunk deployment as plain text or as XML data. In the schema for the modular input, use the <streaming_mode> tag to specify the streaming mode. Specify simple for plain text or xml for XML data.

For example, to specify XML data:

<streaming_mode>xml</streaming_mode>

Simple streaming mode

Simple mode (plain text) is the default streaming mode and is similar to how Splunk software treats data that is streamed from scripted inputs. In simple mode, Splunk software treats the data much like it treats data read from a file. For more information on streaming from scripted inputs, refer to Scripted inputs overview in this manual.

In simple streaming mode, Splunk software supports all character sets described in Configure character set encoding

XML streaming mode

With the Modular Inputs feature, new with Splunk 5.0, there is a new way to stream XML data to the Splunk platform. With this format for streaming XML you can:

• Clearly break events without the use of special markers.
• Easily forward data in a distributed environment by arbitrarily specifying done keys.
• Easily allow a single stream of data to specify source, sourcetype, host, and index.

The format of XML streaming differs, depending on which mode your script specifies:

• one script instance per input stanza mode
• single script instance mode

In XML streaming mode, the XML stream itself must be encoded in UTF-8.

Default parameters when streaming events

The Splunk platform provides default values for the following parameters when streaming events. If Splunk software does not find a definition for these parameters in inputs.conf files, it uses the default values for these parameters.

source

sourcetype

host

index

However, the default value varies, depending on whether you are using one script instance per input stanza mode or single script instance mode. The following table lists the default values for these parameters. The third column of the table lists the default values when using traditional scripted inputs.

Parameter	One script instance per input stanza	Single script instance	Traditional scripted inputs
source	scheme:// (for example, myScheme://abc)	scheme name (for example, myScheme)	script://<path> (<path> = envvar-expanded path from inputs.conf
sourcetype	scheme name	scheme name	exec (or if present, the layered value of the sourcetype)
host	Layered host for each stanza	Global default host from inputs.conf	Layered host from its stanza
index	Layered index for each stanza	Global default index from inputs.conf	Layered index from its stanza

Specify the time of events in the input stream

If an input script knows the time of the event that it generates you can use the <time> tag to specify the time in the input stream. Specify the time using a UTC UNIX timestamp. Subseconds are supported (for example, <time>1330717125.125</time>).

Note: When writing modular input scripts, it is best to specify the time of an event with the tag. Splunk software does not read the timestamp from the body of the event (except in the case of unbroken events, described below). If a tag is not present, Splunk software attempts to use the time the data arrives from the input source as the time of the event.

When specifying the time of events, in props.conf set SHOULD_LINEMERGE to false. Refer to Configure event linebreaking in the Getting Data In manual for more information on setting this property.

Setting SHOULD_LINEMERGE to false does the following:

• Prevents the merging of events because of a missing timestamp.

• Does not override the value set with the <time> tag with a timestamp in the event.

The following example shows how to specify time events in the input stream:

<stream>
  <event stanza="my_config://aaa">
    <time>1330717125</time>
    <data>type=CCC</data>
  </event>
  <event stanza="my_config://bbb">
    <time>1330717125</time>
    <data>type=DDD</data>
  </event>
 . . .
</stream>


# Modify $SPLUNK_HOME/etc/apps/myapp/default/props.conf

[my_config]
SHOULD_LINEMERGE = false

Streaming example (XML mode)

The streaming examples in XML mode in this section illustrate the differences between the following:

• one script instance per input stanza mode
• single script instance mode

The examples also show how you can override the default values for the following parameters:

• source
• sourcetype
• host
• index

Note: For these examples, the introspection scheme enables XML streaming mode, as described in Define a scheme for introspection.

One script instance per input stanza mode

This example shows some example XML that a script can stream to splunkd for indexing, using one script instance per input stanza mode. In this mode, there is a separate instance of the script for each input stanza in inputs.conf configuration files.

<stream>
  <event>
    <time>1370031029</time>
    <data>event_status="(0)The operation completed successfully."</data>
  </event>
  <event>
    <time>1370031031</time>
    <data>event_status="(0)The operation completed successfully."</data>
  </event>
</stream>

In this example, the tags clearly delineate the events. This effectively line-breaks the events without any line-breaking configuration.

The values for source, sourcetype, host, and index are the default values, as described in Default parameters when streaming events. You can override the default values by including the new values in the event. The following example specifies custom values for source and index:

<stream>
  <event>
    <time>1370031035</time>
    <data> event_status="(0)The operation completed successfully."</data>
    <source>my_source</source>
    <index>test1</index>
  </event>
</stream>

Note: Subsequent events can specify new values for the source and index parameters, or simply use the default values.

Single script instance mode

This example shows some example XML that a script can stream to splunkd for indexing, using single script instance mode. In this mode, there is only a single instance of the script.

Note: Because you are using a single instance of the script, use the stanza attribute to the <event> tag to specify the stanza for each event. Specifying the stanza attribute is not needed when streaming in one script instance per input stanza mode.

<stream>
  <event stanza="my_config://aaa">
    <time>1370031041</time>
    <data> event_status="(0)The operation completed successfully."</data>
    <host>my_host</host>
  </event>
</stream>

In this example, the value of stanza should be an existing stanza name from inputs.conf that the event belongs to. If the stanza name is not present (or refers to a non-existant stanza name in the conf file) then Splunk software automatically sets the parameters for source, sourcetype, host, and index.

This example overrides the default value for the host parameter.

Stream unbroken events in XML

The XML streaming examples in the previous sections use the tag to delineate, or break, separate events. However, often when you stream data to the Splunk platform, you do not want to break events, and instead let Splunk software interpret the events. You typically send unbroken data in chunks and let Splunk software apply line breaking rules.

You may want to stream unbroken events either because you are streaming a known format to the Splunk platform, or you may not know the format of the data and you want Splunk software to interpret it. The S3 example in this document streams unbroken events in XML mode.

Use the <time> tag when possible

When streaming unbroken events, Splunk software attempts to read timestamps from the body of the events, and break the event based on the timestamps. However, if known, the tag should be provided for unbroken events. When the unbroken segments are merged, the value from the first tag is used. However, it may be overridden by any timestamp extraction rules for the sourcetype.

Use the <done> tag with unbroken events

Use the <done> tag to denote an end of a stream with unbroken events. The <done> tag tells Splunk software to flush the data from its buffer rather than wait for more data before processing it. For example, Splunk software may buffer data that it has read, waiting for a newline character before processing the data. This prevents the data from being indexed until the newline character is read. If you want the data indexed without the newline character, then send the <done> tag.

Specify the unbroken attribute to the <event> tag. Then after you have reached the end of the data you are sending in chunks, send the <done/> tag as indicated in the following example.

<stream>
  <event unbroken="1">
    <data>09/08/2009 14:01:59.0398 part of the event ...</data>
  </event>
  <event unbroken="1">
    <data>final part of the event</data>
    <done/>
  </event>
  <event unbroken="1">
    <data>second event</data>
    <done/>
  </event>
</stream>

When sending unbroken events:

• You can specify source, sourcetype, host, index, and stanza specifications just as you would when sending broken events.

• The script is responsible for sending a <done/> tag. This is important for forwarders because they can't switch a stream until they see a <done/> tag.

• When the data goes through the time extraction process, if a subset of the event is identified as a timestamp, that time becomes the event's time, and the timestamp is used for event aggregation. Refer to Configure event linebreaking for more information.