Ingest pcap files
Splunk Stream provides the following methods for ingesting pcap file data:
- Upload pcaps in Splunk Web
- Ingest pcaps using command line options
- Ingest pcaps using
streamfwd.conf
Supported file formats
Stream supports both .pcap
and .pcapng
file formats on Linux and Mac.
Stream does not support .pcapng
file format on Windows. To use .pcapng
files on Windows, you must first convert them to .pcap
file format.
Upload pcaps in Splunk Web
You can upload and index pcap file data using modular inputs in Splunk Web.
- In Splunk Web, go to Settings > Data Inputs.
- Click on PCAP Files > New.
- Configure the modular input:
Field Description Name The name of the PCAP file modular input. PCAP File Click Choose File and select the PCAP file you want to upload. System Time Uses the system time clock as timestamp for each packet read. Repeat Continuously repeats the PCAP file until the streamfwd
process is terminated.Host field value The name of the host that will appear in PCAP events. Index The destination index for the PCAP file data. - Click Next.
The pcap file data is uploaded and sent to the specified index.
Ingest pcaps using command line options
Use the following syntax to read pcap file data and send that data to Splunk indexers streamfwd
:
streamfwd [-r FILE1]... [--pcapdir DIR1]... [pcap_options] [options] [output_option]
Note: Relative files or directories are relative to the current working directory.
For detailed streamfwd
command line options and usage information, see streamfwd command line options in this manual.
Read pcap files
Use the -r
option to read individual pcap files. For example:
./streamfwd -r my.pcap
Ingest pcap files from a directory
Use the --pcapdir DIR
option to monitor and index pcap files in a directory. For example:
./streamfwd --pcapdir ~/test_pcap_dir --afteringest repeat
The output behavior behavior of the streamfwd
command differs depending on whether you use an independent streamfwd
deployment or Splunk_TA_stream
. If you use an independent streamfwd
deployment, the output is sent to indexers by HTTP event collector. For Splunk_TA_stream
the output is forwarded to indexers by the Wire data modular input. For more information, see streamfwd command line options.
Note: When ingesting pcaps from directories, make sure the pcap is complete before moving the file to the directory, otherwise the file data will be truncated. Optionally, use a different file extension name (such as .temp) until the pcap file is complete, then change the extension name to .pcap.
Ingest pcaps using streamfwd.conf
You can configure Stream forwarder to ingest individual pcap files, monitor and ingest pcap files from specific directories, monitor live network traffic, or any combination of these simultaneously. Add streamfwdcapture
parameters to streamfwd.conf
.
Use the following streamfwdcapture
parameters to configure pcap file ingestion in streamfwd.conf
:
Parameter | Description |
---|---|
streamfwdcapture.<N>.interface | Specify a network interface name or a path to a pcap file or a directory of pcap files. |
streamfwdcapture.<N>.interfaceRegex | A regular expression specifying which network interfaces to capture. |
streamfwdcapture.<N>.offline | *True means use pcap files: interface must be a pcap file or a directory to monitor for pcap files. *False means interface is a network device name. *Default is false. |
streamfwdcapture.<N>.repeat | *Only applies if interface is a pcap file. *True means to play back the pcap file repeatedly for continuous load. |
streamfwdcapture.<N>.afterIngest | *Only applies if interface is a directory. *Specifies action to take after ingesting a pcap file from the directory. *Possible values:
|
streamfwdcapture.<N>.sysTime
|
*Only applies if offline is true. *True means to use the system time for packet timestamps instead of actual timestamps from pcap file. Default is false. |
streamfwdcapture.<N>.bitsPerSecond
|
*Only applies if offline is true. *Rate limiter: if undefined, defaults to 10 Mbps if <Repeat> is true, else 100 Mbps. |
Note: The streamfwdcapture.<N>.interface
parameter supports both absolute and relative directories. Relative directories are relative to either Splunk_TA_stream/default
or streamfwd/default
.
Examples
The following examples show a variety of streamfwd.conf
configurations using the streamfwdcapture
parameter to ingest pcap files.
Example 1. Ingest a single pcap file
To ingest a pcap file /tmp/server1.pcap
and repeat indefinitely, add the following parameters:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = /tmp/server1.pcap streamfwdcapture.0.repeat = true
where streamfwdcapture.0.offline = true
enables pcap ingestion.
Example 2. Monitor a single directory
To monitor and ingest pcap files in a single directory, in this case /tmp/test_pcap_dir
, add the following parameters:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = /tmp/test_pcap_dir
where streamfwdcapture.0.offline = true
enables pcap ingestion.
Note: If an afterIngest
parameter is not specified, the move
option is used by default. This automatically moves the pcap to the ./finished_pcap
subdirectory after pcap ingestion.
Example 3. Monitor multiple directories
To monitor and ingest pcap files from two or more directories, use multiple streamfwdcapture.<N>
groups. Each directory can have different options, as shown in this example:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = C:\temp\pcap_dir_1 streamfwdcapture.0.sysTime = true streamfwdcapture.1.offline = true streamfwdcapture.1.interface = C:\temp\pcap_dir_2 streamfwdcapture.1.afterIngest = delete
Because streamfwdcapture.0.sysTime
applies only to C:\temp\pcap_dir_1
, pcap
files from C:\temp\pcap_dir_2
are processed using their original timestamps. Similarly, while files from C:\temp\pcap_dir_2
are deleted after ingestion, those from C:\temp\pcap_dir_1
are moved to C:\temp\pcap_dir_1\finished_pcaps
, since that is the default behavior.
Example 4. Monitor both network interface and directory
To capture traffic from a live interface while simultaneously monitoring and ingesting pcap
files from a specified directory, use multiple streamfwdcapture.<N>
groups. For example:
streamfwdcapture.0.offline = false streamfwdcapture.1.offline = true streamfwdcapture.1.interface = /tmp/test_pcap_dir
where streamfwdcapture.0.offline = false
enables monitoring on all available network interfaces (since streamfwdcapture.0.interface
is not specified) and streamfwdcapture.1.offline = true
enables pcap
ingestion from /tmp/test_pcap_dir
directory.
For more information on specifying network interfaces, see Use streamfwdcapture to specify network interfaces in this manual.
Configure universal forwarder for Splunk_TA_stream | Use Splunk Stream to ingest Netflow and IPFIX data |
This documentation applies to the following versions of Splunk Stream™: 7.1.2, 7.1.3, 7.2.0
Feedback submitted, thanks!