dedup command usage
Avoid using the
dedup command on the
_raw field if you are searching over a large volume of data. If you search the
_raw field, the text of every event in memory is retained which impacts your search performance. This is expected behavior. This performance behavior also applies to any field with high cardinality and large size.
Differences between SPL and SPL2
Command options must be specified first
In SPL2, command options must be specified before the <field-list>.
|SPL||... dedup host source 2|
|SPL2||... dedup 2 host, source|
List of fields must be comma-delimited
In SPL2, the list of fields must be comma-delimited. Otherwise a parsing error is returned.
|SPL||... dedup host source|
|SPL2||... dedup host, source|
The sortby argument is not supported
sortby argument is not supported in SPL2. Use the
sort command before the
dedup command if you want to change the order of the events, which dictates which event is kept when the
dedup command is run.
|SPL||... dedup host source sortby -_size|
|SPL2||... sort -_size | dedup host, source|
Alternative: If you are using the
from command, you can specify the
ORDER BY clause instead of using the
The keepevents argument is not supported
keepevents=<boolean> argument is not supported in SPL2.
|SPL||... dedup host keepevents=true|
dedup command syntax details
dedup command examples
This documentation applies to the following versions of Splunk® Cloud Services: current