dedup command overview

The SPL2 dedup command removes the events that contain an identical combination of values for the fields that you specify.

With the SPL2 dedup command, you can specify the number of duplicate events to keep for each value of a single field, or for each combination of values among several fields.

Events returned by the dedup command are based on search order. For historical searches, the most recent events are searched first. For real-time searches, the first events that are received are searched, which are not necessarily the most recent events.

Syntax

The required syntax is in bold.

dedup

[<int>]

[keepempty=<bool>]

[consecutive=<bool>]

<field-list>

How the SPL2 dedup command works

Suppose that you have the following search results:

host	action	client_ip
www1	view	211.166.11.101
www2	addtocart	194.215.205.19
www3	view	74.53.23.135
www1	addtocart	128.241.220.82
www1	purchase	64.66.0.20
www3	view	107.3.146.207
www2	remove	194.215.205.19

You want to remove search results where the host is a duplicate value.

... | dedup host

The results show the unique host names.

host	action	client_ip
www1	view	211.166.11.101
www2	addtocart	194.215.205.19
www3	view	74.53.23.135

This example returns only one result for each host value.

You can specify more than one field with the SPL2 dedup command. For example:

... | dedup host, client_ip

For each combination of host name and client IP address, duplicate results are removed.

host	action	client_ip
www1	view	211.166.11.101
www2	addtocart	194.215.205.19
www3	view	74.53.23.135
www1	addtocart	128.241.220.82
www1	purchase	64.66.0.20
www3	view	107.3.146.207

In this example, the result with host=www2 and client_ip=194.215.205.19 is removed.

See also

dedup command

dedup command syntax details

dedup command usage

dedup command examples