Search Reference



NOTE - Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.



Removes the events which contain an identical combination of values for selected fields.


dedup [<N>] <field-list> [keepevents=<bool>] [keepempty=<bool>] [consecutive=<bool>] [sortby <sort-by-clause>]

Required arguments

Syntax: <string> <string> ...
Description: A list of field names.

Optional arguments

Syntax: consecutive=<bool>
Description: If true, only remove events with duplicate combinations of values that are consecutive. Defaults to false.
Syntax: keepempty=<bool>
Description: If true, keep event events where one or more of the selected fields is null (not present). By default (false), all events where any of the selected fields are null are dropped.
Syntax: keepevents=<bool>
Description: If true, keep all events, but will remove the selected fields from events after the first event containing a particular combination of values. Defaults to false, meaning to drop events after the first event of each particular combination.
Syntax: <int>
Description: As an alternative to retaining only one event for each combination of field values, dedup can retain multiple events for each combination. The number N selects how many events for each combination will be retained. This number must be greater than 0.
Syntax: ( - | + ) <sort-field> [(- | +) <sort_field> ...]
Description: List of fields to sort by and their order, descending ( - ) or ascending ( + ) for each field.

Sort field options

Syntax: <field> | auto(<field>) | str(<field>) | ip(<field>) | num(<field>)
Description: Options for sort-field.
Syntax: <string>
Description: The name of the field to sort.
Syntax: auto(<field>)
Description: Determine automatically how to sort the field's values.
Syntax: ip(<field>)
Description: Interpret the field's values as an IP address.
Syntax: num(<field>)
Description: Treat the field's values as numbers.
Syntax: str(<field>)
Description: Order the field's values lexicographically.


The dedup command lets you specify a number of duplicate events to keep for each value of a single field, or for each combination of values among several fields. Events returned by dedup will be the first event found in search order, which is the most recent for historical search, and the first to arrive for realtime search. If you specify a leading number, N, dedup interprets this number as the quantity events with duplicate values or value combinations to keep. If you don't specify a number, only the first occurring event will be kept, and drops all duplicates.

The dedup command also lets you sort by a list of fields. In this case, dedup deduplicate the results based on the specified sort-by field or fields. The other options will allow you to retain events but with the duplicate fields removed, or keep events where the fields do not exist.

Note that keepempty=true will counter-intuitively keep every event that lacks any of the fields in the field list, aka has a null value for any of the fields. If you want to keep N representative events for combinations of field values including null values, use fillnull to provide a non-null value for these fields. For example:
...|fillnull value="MISSING" field1 field2 | dedup field1 field2

Note: We do not recommend that you run the dedup command against the _raw field if you are searching over a large volume of data. Doing this forces Splunk to retain the text of every event in memory which will impact your search performance. This is expected behavior. This applies to any field with high cardinality and large size.


Example 1: Remove duplicates of results with the same 'host' value.

... | dedup host

Example 2: Remove duplicates of results with the same 'source' value and sort the events by the '_time' field in ascending order.

... | dedup source sortby +_time

Example 3: Remove duplicates of results with the same 'source' value and sort the events by the '_size' field in descending order.

... | dedup source sortby -_size

Example 4: For events that have the same 'source' value, keep the first 3 that occur and remove all subsequent events.

... | dedup 3 source

Example 5: For events that have the same 'source' AND 'host' values, keep the first 3 that occur and remove all subsequent events.

... | dedup 3 source host

See also



Have questions? Visit Splunk Answers and see what questions and answers the Splunk community has using the dedup command.

This documentation applies to the following versions of Splunk: 4.1 , 4.1.1 , 4.1.2 , 4.1.3 , 4.1.4 , 4.1.5 , 4.1.6 , 4.1.7 , 4.1.8 , 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 , 4.3.3 , 4.3.4 , 4.3.5 , 4.3.6 , 4.3.7 , 5.0 , 5.0.1 , 5.0.2 , 5.0.3 , 5.0.4 , 5.0.5 , 5.0.6 , 5.0.7 , 5.0.8 , 5.0.9 , 5.0.10 , 5.0.11 , 5.0.12 , 6.0 , 6.0.1 , 6.0.2 , 6.0.3 , 6.0.4 , 6.0.5 , 6.0.6 , 6.0.7 , 6.0.8 , 6.1 , 6.1.1 , 6.1.2 , 6.1.3 , 6.1.4 , 6.1.5 , 6.1.6 , 6.1.7 , 6.2.0 , 6.2.1 , 6.2.2 , 6.2.3 View the Article History for its revisions.


I carried out a simple test and discovered that to dedup and keep the LATEST event, you want to "| dedup field1 sortby -_time"

Bedgar oneok
December 26, 2014

We need an answer to Landen99's question. Does "| dedup field1 sortby +_time" keep the earliest matching event or the latest matching event?

Bedgar oneok
December 26, 2014

Question on the "sortby" option. Does "sortby" sort the results of the dedup or sort for the dedup? In other words:<br /><br /> | dedup field1 sortby +_time<br /><br />1) the first event for each value is chosen and then these events are sorted by time OR<br />2) the earliest event is chosen for each value?

November 26, 2014

You must be logged into in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!