Splunk® Cloud Services

SPL2 Search Reference

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

dedup command examples

The following are examples for using the SPL2 dedup command. To learn more about the dedup command, see How the dedup command works.

1. Remove duplicate results based on one field

Remove duplicate search results with the same host value.

... | dedup host

2. Keep the first 3 duplicate results

For search results that have the same source value, keep the first 3 that occur and remove all subsequent results.

... | dedup 3 source

3. Sort events in ascending order before removing duplicate values

Use the order by clause in the from command to sort the events by time in ascending order, the default order. Sorting the events ensures that the oldest events are listed first. Remove duplicate results with the same source value. Only the oldest events are retained.

| from main order by ASC _time | dedup source

4. Sort events after removing duplicate values

Remove duplicate search results with the same host value and sort the events by the _size field in descending order.

... | dedup host | sort -_size

5. Keep results that have the same combination of values in multiple fields

For search results that have the same combination of source AND host values, keep the first 2 that occur and remove all subsequent results.

... | dedup 2 source, host

6. Remove only consecutive duplicate events

Remove only consecutive duplicate events. Keep non-consecutive duplicate events. In this example duplicates must have the same combination of values the source and host fields.

... | dedup consecutive=true source, host

See also

dedup command
dedup command overview
dedup command syntax details
dedup command usage
Last modified on 29 April, 2020
PREVIOUS
dedup command usage
  NEXT
eval command overview

This documentation applies to the following versions of Splunk® Cloud Services: current


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters