Splunk® Cloud Services

SPL2 Search Reference

dedup command usage

Avoid using the dedup command on the _raw field if you are searching over a large volume of data. If you search the _raw field, the text of every event in memory is retained which impacts your search performance. This is expected behavior. This performance behavior also applies to any field with high cardinality and large size.

Differences between SPL and SPL2

Command options must be specified first

In SPL2, command options must be specified before the <field-list>.

Version Example
SPL ... dedup host source 2
SPL2 ... dedup 2 host, source

List of fields must be comma-delimited

In SPL2, the list of fields must be comma-delimited. Otherwise a parsing error is returned.

Version Example
SPL ... dedup host source
SPL2 ... dedup host, source

The sortby argument is not supported

The sortby argument is not supported in SPL2. Use the sort command before the dedup command if you want to change the order of the events, which dictates which event is kept when the dedup command is run.

Version Example
SPL ... dedup host source sortby -_size
SPL2 ... sort -_size | dedup host, source

Alternative: If you are using the from command, you can specify the ORDER BY clause instead of using the sort command.

The keepevents argument is not supported

The keepevents=<boolean> argument is not supported in SPL2.

Version Example
SPL ... dedup host keepevents=true
SPL2 Not supported

See also

dedup command
dedup command overview
dedup command examples
Last modified on 29 April, 2020
dedup command syntax details   dedup command examples

This documentation applies to the following versions of Splunk® Cloud Services: current

Was this topic useful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters