dedup command usage

Avoid using the dedup command on the _raw field if you are searching over a large volume of data. If you search the _raw field, the text of every event in memory is retained which impacts your search performance. This is expected behavior. This performance behavior also applies to any field with high cardinality and large size.

Differences between SPL and SPL2

Command options must be specified first

In SPL2, command options must be specified before the <field-list>.

Version	Example
SPL	... dedup host source 2
SPL2	... dedup 2 host, source

List of fields must be comma-delimited

In SPL2, the list of fields must be comma-delimited. Otherwise a parsing error is returned.

Version	Example
SPL	... dedup host source
SPL2	... dedup host, source

The sortby argument is not supported

The sortby argument is not supported in SPL2. Use the sort command before the dedup command if you want to change the order of the events, which dictates which event is kept when the dedup command is run.

Version	Example
SPL	... dedup host source sortby -_size
SPL2	... sort -_size | dedup host, source

Alternative: If you are using the from command, you can specify the ORDER BY clause instead of using the sort command.

The keepevents argument is not supported

The keepevents=<boolean> argument is not supported in SPL2.

Version	Example
SPL	... dedup host keepevents=true
SPL2	Not supported

See also

dedup command

dedup command overview

dedup command examples