Ingest pcap files

Splunk Stream provides the following methods for ingesting pcap file data:

• Upload pcaps in Splunk Web.
• Ingest pcaps using command line options.
• Ingest pcaps using streamfwd.conf.

Supported file formats

Stream supports both .pcap and .pcapng file formats on Linux and Mac.

Stream does not support .pcapng file format on Windows. To use .pcapng files on Windows, you must first convert them to .pcap file format.

Upload pcaps in Splunk Web

You can upload and index pcap file data using modular inputs in Splunk Web.

1. In Splunk Web, go to Settings > Data Inputs.
2. Click on PCAP Files > New.
3. Configure the modular input:

   Field	Description
   Name	The name of the PCAP file modular input.
   PCAP File	Click Choose File and select the PCAP file you want to upload.
   System Time	Uses the system time clock as timestamp for each packet read.
   Repeat	Continuously repeats the PCAP file until the streamfwd process is terminated.
   Host field value	The name of the host that will appear in PCAP events.
   Index	The destination index for the PCAP file data.

4. Click Next.
   The pcap file data is uploaded and sent to the specified index.

Ingest pcaps using command line options

You can use streamfwd command line options to read pcap file data and send that data to Splunk indexers, using the following syntax:

streamfwd [-r FILE1]... [--pcapdir DIR1]... [pcap_options] [options] [output_option]

Note: Relative files or directories are relative to the current working directory.

For detailed streamfwd command line options and usage information, see streamfwd command line options in this manual.

Read pcap files

Use the -r option to read individual pcap files. For example:

./streamfwd -r my.pcap

Ingest pcap files from a directory

Use the --pcapdir DIR option to monitor and index pcap files in a directory. For example:

./streamfwd --pcapdir ~/test_pcap_dir --afteringest repeat

The output behavior behavior of the streamfwd command differs depending on whether you are using an independent streamfwd deployment or Splunk_TA_stream. If you are using an independent streamfwd deployment, the output is sent to indexers by HTTP event collector. If you are using Splunk_TA_stream the output is forwarded to indexers by the Wire data modular input. For more information, see streamfwd command line options.

Note: When ingesting pcaps from directories, make sure the pcap is complete before moving the file to the directory, otherwise the file data will be truncated. Optionally, use a different file extension name (such as .temp) until the pcap file is complete, then change the extension name to .pcap.

Ingest pcaps using streamfwd.conf

You can configure Stream forwarder to ingest individual pcap files, monitor and ingest pcap files from specific directories, monitor live network traffic, or any combination of these simultaneously. This is done by adding streamfwdcapture parameters to streamfwd.conf.

Use the following streamfwdcapture parameters to configure pcap file ingestion in streamfwd.conf:

Parameter	Description
streamfwdcapture.<N>.interface	Specify a network interface name or a path to a pcap file or a directory of pcap files.
streamfwdcapture.<N>.interfaceRegex	A regular expression specifying which network interfaces to capture.
streamfwdcapture.<N>.offline	*True means use pcap files: interface must be a pcap file or a directory to monitor for pcap files. *False means interface is a network device name. *Default is false.
streamfwdcapture.<N>.repeat	*Only applies if interface is a pcap file. *True means to play back the pcap file repeatedly for continuous load.
streamfwdcapture.<N>.afterIngest	*Only applies if interface is a directory. *Specifies action to take after ingesting a pcap file from the directory. *Possible values: delete: Delete the file. move <subdir>: Move the file to a subdirectory (which will be created if needed). Default is finished_pcaps. ignore: Leave the file but mark it as already processed. repeat: Continue to re-ingest all pcap files in rotation. stop: Leave the file. After processing each directory once, stop monitoring. *Default is move.
streamfwdcapture.<N>.sysTime	*Only applies if offline is true. *True means to use the system time for packet timestamps instead of actual timestamps from pcap file. Default is false.
streamfwdcapture.<N>.bitsPerSecond	*Only applies if offline is true. *Rate limiter: if undefined, defaults to 10 Mbps if <Repeat> is true, else 100 Mbps.

Note: The streamfwdcapture.<N>.interface parameter supports both absolute and relative directories. Relative directories are relative to either Splunk_TA_stream/default or streamfwd/default.

Examples

The following examples show a variety of streamfwd.conf configurations using the streamfwdcapture parameter to ingest pcap files.

Example 1. Ingest a single pcap file

To ingest a pcap file /tmp/server1.pcap and repeat indefinitely, add the following parameters:

[streamfwd]
streamfwdcapture.0.offline = true
streamfwdcapture.0.interface = /tmp/server1.pcap
streamfwdcapture.0.repeat = true

where streamfwdcapture.0.offline = true enables pcap ingestion.

Example 2. Monitor a single directory

To monitor and ingest pcap files in a single directory, in this case /tmp/test_pcap_dir, add the following parameters:

[streamfwd]
streamfwdcapture.0.offline = true
streamfwdcapture.0.interface = /tmp/test_pcap_dir

where streamfwdcapture.0.offline = true enables pcap ingestion.

Note: If an afterIngest parameter is not specified, the move option is used by default. This automatically moves the pcap to the ./finished_pcap subdirectory after pcap ingestion.

Example 3. Monitor multiple directories

To monitor and ingest pcap files from two or more directories, use multiple streamfwdcapture.<N> groups. Each directory can have different options, as shown in this example:

[streamfwd]
streamfwdcapture.0.offline = true
streamfwdcapture.0.interface = C:\temp\pcap_dir_1
streamfwdcapture.0.sysTime = true

streamfwdcapture.1.offline = true
streamfwdcapture.1.interface = C:\temp\pcap_dir_2
streamfwdcapture.1.afterIngest = delete

Since streamfwdcapture.0.sysTime applies only to C:\temp\pcap_dir_1, pcap files from C:\temp\pcap_dir_2 will be processed using their original timestamps. Similarly, while files from C:\temp\pcap_dir_2 will be deleted after ingestion, those from C:\temp\pcap_dir_1 will be moved to C:\temp\pcap_dir_1\finished_pcaps, since that is the default behavior.

Example 4. Monitor both network interface and directory

To capture traffic from a live interface while simultaneously monitoring and ingesting pcap files from a specified directory, use multiple streamfwdcapture.<N> groups. For example:

streamfwdcapture.0.offline = false

streamfwdcapture.1.offline = true
streamfwdcapture.1.interface = /tmp/test_pcap_dir

where streamfwdcapture.0.offline = false enables monitoring on all available network interfaces (since streamfwdcapture.0.interface is not specified) and streamfwdcapture.1.offline = true enables pcap ingestion from /tmp/test_pcap_dir directory.

For more information on specifying network interfaces, see Use streamfwdcapture to specify network interfaces in this manual.