Splunk® Enterprise

Forwarding Data

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Forward data to third-party systems

Splunk Enterprise forwarders can forward raw data to non-Splunk systems. They can send the data over a plain TCP socket or packaged in standard syslog. Because they are forwarding to a non-Splunk system, they can send only raw data.

By editing outputs.conf, props.conf, and transforms.conf, you can configure a heavy forwarder to route data conditionally to third-party systems, in the same way that it routes data conditionally to other Splunk Enterprise instances. You can filter the data by host, source, or source type. You can also use regex to further qualify the data.

TCP data

To forward TCP data to a third-party system, edit the forwarder's outputs.conf file to specify the receiving server and port. You must also configure the receiving server to expect the incoming data stream on that port. You can use any kind of forwarder, such as a universal forwarder, to perform this type of forwarding.

To route the data, you need to use a heavy forwarder, which has the ability to parse data. Edit the forwarder's props.conf and transforms.conf files as well as outputs.conf.

Edit the configuration files

To simply forward data, edit outputs.conf:

  • Specify target groups for the receiving servers.
  • Specify the IP address and TCP port for each receiving server.
  • Set sendCookedData to false, so that the forwarder sends raw data.

To route and filter the data (heavy forwarders only), also edit props.conf and transforms.conf:

  • In props.conf, specify the host, source, or sourcetype of your data stream. Specify a transform to perform on the input.
  • In transforms.conf, define the transform and specify _TCP_ROUTING. You can also use regex to further filter the data.

Forward all data

This example shows how to send all the data from a universal forwarder to a third-party system. Since you are sending all the data, you only need to edit outputs.conf:

[tcpout]

[tcpout:fastlane]
server = 10.1.1.35:6996
sendCookedData = false

Forward a subset of data

This example shows how to use a heavy forwarder to filter a subset of data and send the subset to a third-party system:

1. Edit props.conf and transforms.conf to specify the filtering criteria.

In props.conf, apply the bigmoney transform to all host names beginning with nyc:

[host::nyc*]
TRANSFORMS-nyc = bigmoney

In transforms.conf, configure the bigmoney transform to specify TCP_ROUTING as the DEST_KEY and the bigmoneyreader target group as the FORMAT:

[bigmoney]
REGEX = .
DEST_KEY=_TCP_ROUTING
FORMAT=bigmoneyreader

2. In outputs.conf, define both a bigmoneyreader target group for the non-Splunk server and a default target group to receive any other data:

[tcpout]
defaultGroup = default-clone-group-192_168_1_104_9997

[tcpout:default-clone-group-192_168_1_104_9997]
server = 192.168.1.104:9997

[tcpout:bigmoneyreader]
server=10.1.1.197:7999
sendCookedData=false

The forwarder will send all data from host names beginning with nyc to the non-Splunk server specified in the bigmoneyreader target group. It will send data from all other hosts to the server specified in the default-clone-group-192_168_1_104_9997 target group.

Note: If you want to forward only the data specifically identified in props.conf and transforms.conf, set defaultGroup=nothing.

Syslog data

You can configure a heavy forwarder to send data in standard syslog format. The forwarder sends the data through a separate output processor. You can also filter the data with props.conf and transforms.conf. You'll need to specify _SYSLOG_ROUTING as the DEST_KEY.

Note: The syslog output processor is not available for universal or light forwarders.

The syslog output processor sends RFC 3164 compliant events to a TCP/UDP-based server and port, making the payload of any non-compliant data RFC 3164 compliant. Yes, that means Windows event logs!

To forward syslog data, identify the third-party receiving server and specify it in a syslog target group in the forwarder's outputs.conf file.

Note: If you have defined multiple event types for syslog data, the event type names must all include the string "syslog".

Forward syslog data

In outputs.conf, specify the syslog target group:

[syslog:<target_group>]
<attribute1> = <val1>
<attribute2> = <val2>
...

The target group stanza requires this attribute:

Required Attribute Default Value
server n/a This must be in the format <ipaddress_or_servername>:<port>. This is a combination of the IP address or servername of the syslog server and the port on which the syslog server is listening. Note that syslog servers use port 514 by default.


These attributes are optional:

Optional Attribute Default Value
type udp The transport protocol. Must be set to "tcp" or "udp".
priority <13> - this signifies a facility of 1 ("user") and a severity of 5 ("notice") Syslog priority. This must be an integer 1 to 3 digits in length, surrounded by angle brackets; for example: <34>. This value will appear in the syslog header.

Mimics the number passed via syslog interface call; see outputs.conf for more information.

Compute the priority value as (<facility> * 8) + <severity>. If facility is 4 (security/authorization messages) and severity is 2 (critical conditions), priority value will be: (4 * 8) + 2 = 34, which you specify in the conf file as <34>.

syslogSourceType n/a This must be in the format sourcetype::syslog, the source type for syslog messages.
timestampformat "" The format used when adding a timestamp to the header. This must be in the format: <%b %e %H:%M:%S>. See "Configure timestamps" in the Getting Data In manual for details.


Send a subset of data to a syslog server

This example shows how to configure a heavy forwarder to forward data from hosts whose names begin with "nyc" to a syslog server named "loghost.example.com" over port 514:

1. Edit props.conf and transforms.conf to specify the filtering criteria.

In props.conf, apply the send_to_syslog transform to all host names beginning with nyc:

[host::nyc*]
TRANSFORMS-nyc = send_to_syslog

In transforms.conf, configure the send_to_syslog transform to specify _SYSLOG_ROUTING as the DEST_KEY and the my_syslog_group target group as the FORMAT:

[send_to_syslog]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

2. In outputs.conf, define the my_syslog_group target group for the non-Splunk server:

[syslog:my_syslog_group]
server = loghost.example.com:514
PREVIOUS
Route and filter data
  NEXT
Deploy a heavy forwarder

This documentation applies to the following versions of Splunk® Enterprise: 6.0, 6.0.1, 6.0.2, 6.0.3, 6.0.4, 6.0.5, 6.0.6, 6.0.7, 6.0.8, 6.0.9, 6.0.10, 6.0.11, 6.0.12, 6.0.13, 6.0.14, 6.0.15, 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14


Comments

Tdiestel, Dvb: Could you pose these questions to Splunk Answers? <br /><br />http://answers.splunk.com/answers/index.html<br /><br />That's the best place to get a quick response for these sorts of issues. (And it increases visibility to the community at large.) <br /><br />Thanks.

Sgoodman
September 26, 2014

Can forwarder send data to Splunk AND also the same incoming data to a third party system?

Tdiestel
September 24, 2014

Is there a possibility to forward data to a syslog server _without_ adding a syslog header (e.g. because there already is one that should be kept)?

Dvb
August 13, 2014

Hi Dfsixstring--<br /><br />Unfortunately the solution to your question involves designing a regular expression to extract that field value from your events. If you need help with regular expression design there are a number of non-Splunk websites and apps that can help you. Alternatively you could seek help from the community of Splunk users over at Splunk Answers, where you can find a number of regex wizards looking for easy questions to answer: http://answers.splunk.com/

Mness
July 7, 2014

I'm not a coder of any sort, but I have my Watchguard XTM505 sending syslog data to a Linux box that also has Splunk running on it. I'd like to get some meaningful stats on access attempts [deny], [allow], etc. I don't know how to parse the syslog output into a way that I can use it in splunk. Here's an example of what a deny entry of the syslog looks:<br /><br />Jul 2 15:22:45 XTM505 8XXXXXXXXX0E (2014-07-02T20:22:45) firewall: msg_id="3000-0148" Deny 0-External Firebox 48 tcp 20 114 103.16.34.232 70.XXX.XXX.XXX 1522 445 offset 7 S 2467311678 win 65535 (Unhandled External Packet-00)<br /><br />I'd like to be able to pull out the 103.16.34.232 and run that through a geoIP lookup.

Dfsixstring
July 2, 2014

Vadud3 -<br /><br />You can't do that, but here's a alternative that should work, suggested by a member of our esteemed support team:<br /><br />- outputs.conf<br />[syslog]<br />defaultGroup=nothing<br /><br /><br />[syslog:serverX]<br />server = loghost.example.com:514<br />[syslog:serverY]<br />server = loghost2.example.com:514<br /><br />- props.conf<br />[source::B]<br />TRANSFORMS-routing=syslogRouting<br /><br /><br />- transforms.conf<br />[syslogRouting]<br />REGEX=.<br />DEST_KEY=_SYSLOG_ROUTING<br />FORMAT=serverX,serverY<br /><br />I'll be adding that example into the wiki page proper. Thanks for asking the question.

Sgoodman
January 24, 2014

Can I have multiple syslog servers like below?<br /><br />[syslog:my_syslog_group]<br />server = loghost.example.com:514<br />server = loghost2.example.com:514

Vadud3
January 23, 2014

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters