Ingest pcap files
Splunk Stream provides the following methods for ingesting pcap file data:
- Upload pcaps in Splunk Web.
- Ingest pcaps using command line options.
- Ingest pcaps using streamfwd.conf.
Supported file formats
Stream supports both .pcap and .pcapng file formats on Linux and Mac.
Stream does not support .pcapng file format on Windows. To use .pcapng files on Windows, you must first convert them to .pcap file format.
Upload pcaps in Splunk Web
You can upload and index pcap file data using modular inputs in Splunk Web.
- In Splunk Web, go to Settings > Data Inputs.
- Click on PCAP Files > New.
- Configure the modular input:
Field Description Name The name of the PCAP file modular input. PCAP File Click Choose File and select the PCAP file you want to upload. System Time Uses the system time clock as timestamp for each packet read. Repeat Continuously repeats the PCAP file until the
streamfwdprocess is terminated.
Host field value The name of the host that will appear in PCAP events. Index The destination index for the PCAP file data.
- Click Next.
The pcap file data is uploaded and sent to the specified index.
Ingest pcaps using command line options
You can use
streamfwd command line options to read pcap file data and send that data to Splunk indexers, using the following syntax:
streamfwd [-r FILE1]... [--pcapdir DIR1]... [pcap_options] [options] [output_option]
Note: Relative files or directories are relative to the current working directory.
streamfwd command line options and usage information, see streamfwd command line options in this manual.
Read pcap files
-r option to read individual
pcap files. For example:
./streamfwd -r my.pcap
Ingest pcap files from a directory
--pcapdir DIR option to monitor and index pcap files in a directory. For example:
./streamfwd --pcapdir ~/test_pcap_dir --afteringest repeat
The output behavior behavior of the
streamfwd command differs depending on whether you are using an independent
streamfwd deployment or
Splunk_TA_stream. If you are using an independent
streamfwd deployment, the output is sent to indexers by HTTP event collector. If you are using
Splunk_TA_stream the output is forwarded to indexers by the Wire data modular input. For more information, see streamfwd command line options.
Note: When ingesting pcaps from directories, make sure the pcap is complete before moving the file to the directory, otherwise the file data will be truncated. Optionally, use a different file extension name (such as .temp) until the pcap file is complete, then change the extension name to .pcap.
Ingest pcaps using streamfwd.conf
You can configure Stream forwarder to ingest individual pcap files, monitor and ingest pcap files from specific directories, monitor live network traffic, or any combination of these simultaneously. This is done by adding
streamfwdcapture parameters to
Use the following
streamfwdcapture parameters to configure pcap file ingestion in
|streamfwdcapture.<N>.interface||Specify a network interface name or a path to a pcap file or a directory of pcap files.|
|streamfwdcapture.<N>.interfaceRegex||A regular expression specifying which network interfaces to capture.|
|streamfwdcapture.<N>.offline||*True means use pcap files: interface must be a pcap file or a directory to monitor for pcap files.|
*False means interface is a network device name.
*Default is false.
|streamfwdcapture.<N>.repeat||*Only applies if interface is a pcap file.|
*True means to play back the pcap file repeatedly for continuous load.
|streamfwdcapture.<N>.afterIngest||*Only applies if interface is a directory.|
*Specifies action to take after ingesting a pcap file from the directory.
|streamfwdcapture.<N>.sysTime||*Only applies if offline is true.|
*True means to use the system time for packet timestamps instead of actual timestamps from pcap file. Default is false.
|streamfwdcapture.<N>.bitsPerSecond||*Only applies if offline is true.|
*Rate limiter: if undefined, defaults to 10 Mbps if <Repeat> is true, else 100 Mbps.
streamfwdcapture.<N>.interface parameter supports both absolute and relative directories. Relative directories are relative to either
The following examples show a variety of
streamfwd.conf configurations using the
streamfwdcapture parameter to ingest pcap files.
Example 1. Ingest a single pcap file
To ingest a pcap file
/tmp/server1.pcap and repeat indefinitely, add the following parameters:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = /tmp/server1.pcap streamfwdcapture.0.repeat = true
streamfwdcapture.0.offline = true enables pcap ingestion.
Example 2. Monitor a single directory
To monitor and ingest pcap files in a single directory, in this case
/tmp/test_pcap_dir, add the following parameters:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = /tmp/test_pcap_dir
streamfwdcapture.0.offline = true enables pcap ingestion.
Note: If an
afterIngest parameter is not specified, the
move option is used by default. This automatically moves the pcap to the
./finished_pcap subdirectory after pcap ingestion.
Example 3. Monitor multiple directories
To monitor and ingest pcap files from two or more directories, use multiple
streamfwdcapture.<N> groups. Each directory can have different options, as shown in this example:
[streamfwd] streamfwdcapture.0.offline = true streamfwdcapture.0.interface = C:\temp\pcap_dir_1 streamfwdcapture.0.sysTime = true streamfwdcapture.1.offline = true streamfwdcapture.1.interface = C:\temp\pcap_dir_2 streamfwdcapture.1.afterIngest = delete
streamfwdcapture.0.sysTime applies only to
C:\temp\pcap_dir_1, pcap files from
C:\temp\pcap_dir_2 will be processed using their original timestamps. Similarly, while files from
C:\temp\pcap_dir_2 will be deleted after ingestion, those from
C:\temp\pcap_dir_1 will be moved to
C:\temp\pcap_dir_1\finished_pcaps, since that is the default behavior.
Example 4. Monitor both network interface and directory
To capture traffic from a live interface while simultaneously monitoring and ingesting pcap files from a specified directory, use multiple
streamfwdcapture.<N> groups. For example:
streamfwdcapture.0.offline = false streamfwdcapture.1.offline = true streamfwdcapture.1.interface = /tmp/test_pcap_dir
streamfwdcapture.0.offline = false enables monitoring on all available network interfaces (since
streamfwdcapture.0.interface is not specified) and
streamfwdcapture.1.offline = true enables pcap ingestion from
For more information on specifying network interfaces, see Use streamfwdcapture to specify network interfaces in this manual.
Configure universal forwarder for Splunk_TA_stream
Use Stream configuration templates
This documentation applies to the following versions of Splunk Stream™: 7.1.0, 7.1.1