Ingest PCAP files
Packet Capture (PCAP) is an application interface you can use with Splunk Stream to capture network traffic. Splunk Stream provides several methods for ingesting PCAP file data:
- Upload PCAP data in Splunk Web
- Ingest PCAP files using command line options
- Ingest PCAP files using an Independent Stream Forwarder
Supported file formats
Splunk Stream supports .pcap
and .pcapng
file formats on Linux and Mac.
Splunk Stream does not support .pcapng
file format on Windows. To use .pcapng
files on Windows, you must first convert them to .pcap
file format.
Upload and index PCAP data in Splunk Web
To upload and index PCAP data in Splunk Web, you must have two Splunk Stream packages installed on your search head.
- Splunk App for Stream
splunk_app_stream
- Splunk Add-on for for Stream Forwarder
Splunk_TA_stream
- In Splunk Web, go to Settings > Data Inputs.
- Click on PCAP Files > New.
- Configure the modular input:
Field Description Name The name of the PCAP file modular input. PCAP File Click Choose File and select the PCAP file you want to upload. System Time Uses the system time clock as timestamp for each packet read. Repeat Continuously repeats the PCAP file until the streamfwd
process is terminated.Host field value The name of the host that will appear in PCAP events. Index The destination index for the PCAP file data. - Click Next.
The PCAP file data is uploaded and sent to the specified index.
Ingest PCAP files using command line options
To read PCAP file data and send that data to Splunk indexers, you must have Splunk_TA_stream
installed.
streamfwd [-r FILE1]... [--pcapdir DIR1]... [pcap_options] [options] [output_option]
Note: Relative files or directories are relative to the current working directory.
For detailed streamfwd
command line options and usage information, see streamfwd command line options in this manual.
Read PCAP files
Use the -r
option to read individual PCAP files.
./streamfwd -r my.pcap
Ingest PCAP files from a directory
Use the --pcapdir DIR
option to monitor and index PCAP files in a directory.
./streamfwd --pcapdir ~/test_pcap_dir --afteringest repeat
The output behavior behavior of the command depends on the type of Stream forwarder you use in your configuration.
- When you use an Independent Stream Forwarder, the output is sent to indexers by the HTTP event collector.
- When you use the Splunk Add-on for Stream Forwarders, output is forwarded to indexers using the Splunk Add-on for Stream Wire Data.
For more information, see streamfwd command line options.
Note: When ingesting PCAP files from directories, make sure the PCAP is complete before moving the file to the directory so that file data is not truncated. You can also use a different file extension name (such as .temp) until the PCAP file is complete, then change the extension name to .pcap
.
Ingest PCAP files using an Independent Stream Forwarder
You can configure an Independent Stream Forwarder to:
- Ingest individual PCAP files
- Monitor and ingest PCAP files from specific directories,
- Monitor live network traffic,
- Any combination of these simultaneously.
To configure the independent Stream Forwarder to use PCAP, you add streamfwdcapture
parameters to streamfwd.conf
.
Parameter | Description |
---|---|
streamfwdcapture.<N>.interface
|
Specify a network interface name or a path to a PCAP file or a directory of PCAP files. |
streamfwdcapture.<N>.interfaceRegex
|
A regular expression specifying which network interfaces to capture. |
streamfwdcapture.<N>.offline
|
*True means the Independent Forwarder uses PCAP files: the interface must be a PCAP file or a directory to monitor for PCAP files.
|
streamfwdcapture.<N>.repeat
|
*Only applies if interface is a PCAP file.
|
streamfwdcapture.<N>.afterIngest
|
*Only applies if interface is a directory.
|
streamfwdcapture.<N>.sysTime
|
*Only applies ifstreamfwdcapture.<N>.offline is set to true.
|
streamfwdcapture.<N>.bitsPerSecond
|
*Only applies if offline is true.
*Rate limiter: if undefined, defaults to 10 Mbps if <Repeat> is true, else 100 Mbps. |
Note: The streamfwdcapture.<N>.interface
parameter supports both absolute and relative directories. Relative directories are relative to either Splunk_TA_stream/default
(Splunk App for Stream) or streamfwd/default
(Independent Stream Forwarder).
Examples
The following examples show streamfwd.conf
configurations using the streamfwdcapture
parameter to ingest PCAP files. These examples apply to Splunk Add-on for Stream Forwarders and Independent Stream Forwarders.
Ingest a single PCAP file
To ingest a PCAP file /tmp/server1.pcap
and repeat indefinitely, add the following parameters:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = /tmp/server1.pcap streamfwdcapture.0.repeat = true
Where streamfwdcapture.0.offline = true
enables PCAP ingestion.
Monitor a single directory
To monitor and ingest PCAP files in a single directory, edit /tmp/test_pcap_dir
to add the following parameters:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = /tmp/test_pcap_dir
In this example streamfwdcapture.0.offline = true
enables PCAP ingestion.
Note: If an afterIngest
parameter is not specified, the move
option is used by default. This automatically moves the PCAP to the ./finished_pcap
subdirectory after PCAP ingestion.
Monitor multiple directories
To monitor and ingest PCAP files from two or more directories, use multiple streamfwdcapture.<N>
groups. Each directory can have different options:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = C:\temp\pcap_dir_1 streamfwdcapture.0.sysTime = true streamfwdcapture.1.offline = true streamfwdcapture.1.interface = C:\temp\pcap_dir_2 streamfwdcapture.1.afterIngest = delete
In this example:
pcap
files fromC:\temp\pcap_dir_2
are processed using their original timestamps becausestreamfwdcapture.0.sysTime
applies only toC:\temp\pcap_dir_1
.- Files from
C:\temp\pcap_dir_2
are deleted after ingestion. - Files from
C:\temp\pcap_dir_1
are moved toC:\temp\pcap_dir_1\finished_pcaps
.
Monitor both network interface and directory
To capture traffic from a live interface while simultaneously monitoring and ingesting pcap
files from a specified directory, use multiple streamfwdcapture.<N>
groups. For example:
streamfwdcapture.0.offline = false streamfwdcapture.1.offline = true streamfwdcapture.1.interface = /tmp/test_pcap_dir
In this example:
streamfwdcapture.0.offline = false
enables monitoring on all available network interfaces becausestreamfwdcapture.0.interface
is not specifiedstreamfwdcapture.1.offline = true
enablespcap
ingestion from/tmp/test_pcap_dir
directory.
For more information about specifying network interfaces, see Use streamfwdcapture to specify network interfaces in this manual.
Use SSL keys for decryption on forwarders | Use Splunk Stream to ingest Netflow and IPFIX data |
This documentation applies to the following versions of Splunk Stream™: 8.0.1, 8.0.2, 8.1.0, 8.1.1, 8.1.3
Feedback submitted, thanks!