Splunk® Enterprise

Admin Manual

Download manual as PDF

Splunk version 4.x reached its End of Life on October 1, 2013. Please see the migration information.
This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Remove indexed data from Splunk

You can remove data from indexes in these ways:

  • Delete events from future searches with the delete operator.
  • Remove all data from one or more indexes with the CLI clean command.
  • Delete an entire index.
  • Delete older data, based on a retirement policy.

Caution: Removing data is irreversible. Use caution when choosing what events to remove from searches, or what data to remove from your Splunk indexes. If you want to get your data back, you must re-index the applicable data source(s).

Delete data from future searches

Splunk provides the special operator delete to delete data from future searches. Before using delete , read this section carefully.

Note: Splunk does not let you run the delete operator during a real-time search; you cannot delete events as they come in. If you try to use delete during a real-time search, Splunk will display an error.

Who can delete?

The delete operator can only be accessed by a user with the "delete_by_keyword" capability. By default, Splunk ships with a special role, "can_delete" that has this capability (and no others). The admin role does not have this capability by default. Splunk recommends you create a special user that you log into when you intend to delete index data.

For more information, refer to "Add and edit roles" in this manual.

How to delete

To use the delete operator, run a search that returns the events you want deleted. Make sure that this search returns ONLY events you want to delete, and no other events.

For example, if you want to remove the events you've indexed from a source called /fflanda/incoming/cheese.log so that they no longer appear in search results, do the following:

1. Disable or remove that source so that it no longer gets indexed.

2. Search for events from that source in your index:

source="/fflanda/incoming/cheese.log"

3. Look at the results to confirm that this is the data you want to delete.

4. Once you've confirmed that this is the data you want to delete, pipe the search to delete:

source="/fflanda/incoming/cheese.log" | delete

See the page about the delete operator in the Search Reference Manual for more examples.

Piping a search to the delete operator marks all the events returned by that search so that future searches do not return them. No user (even with admin permissions) will be able to see this data when searching with Splunk.

Note: Piping to delete does not reclaim disk space.

The delete operator also does not update the metadata of the events, so any metadata searches will still include the events although they are not searchable. The main All indexed data dashboard will still show event counts for the deleted sources, hosts, or sourcetypes.

Remove data from indexes with the CLI clean command

To delete index data permanently from your disk, use the CLI clean command. This command completely deletes the data in one or all indexes, depending on whether you provide an <index_name> argument. Typically, you run clean before re-indexing all your data.

Note: The CLI clean command is available in all versions of Splunk, including versions for Windows. When issuing CLI commands in Splunk for Windows, simply substitute the forward slashes you see in the examples shown below (/) with backslashes (\).

How to use the clean command

Here are the main ways to use the clean command:

  • To access the help page for clean, type:
      ./splunk help clean
  • To permanently remove event data from all indexes, type:
      ./splunk clean eventdata
  • To permanently remove event data from a single index, type:
      ./splunk clean eventdata -index <index_name>
      where <index_name> is the name of the targeted index. 
  • Add the -f parameter to force clean to skip its confirmation prompts.

Important: You must stop Splunk before you run the clean command:

./splunk stop

Examples

This example removes event data from all indexes:

./splunk clean eventdata 

This example removes event data from the _internal index and forces Splunk to skip the confirmation prompt:

./splunk clean eventdata -index _internal -f

Delete an index entirely

To completely delete an index (and not just the data contained in it), edit indexes.conf and remove its stanza.

Here are the main steps:

1. Look through all inputs.conf files (on your indexer and on any forwarders sending data to the indexer) and make sure that none of the stanzas are directing data to the index you want to delete. In other words, if you want to delete an index called "nogood", make sure the following attribute/value pair does not appear in any of your input stanzas: index=nogood.

2. Stop the indexer.

3. Edit indexes.conf and remove the entire stanza for the index you want to delete.

4. Start the indexer.

Delete older data based on retirement policy

When data in an index reaches a configurable age or when the index grows to a configurable size, it rolls to the "frozen" state, at which point Splunk deletes it from the index. Just before deleting the data, Splunk can move it to an archive, depending on how you configure your retirement policy.

For more information, refer to "Set a retirement and archiving policy" in this manual.

PREVIOUS
Move the index database
  NEXT
Optimize indexes

This documentation applies to the following versions of Splunk® Enterprise: 4.3, 4.3.1, 4.3.2, 4.3.3, 4.3.4, 4.3.5, 4.3.6, 4.3.7


Comments

Jnilson - As you noted, the delete command does not actually delete the data from the index, and the clean command can only be used to delete all data from an index. If you need better granularity, here's some information in Splunk Answers that might be of help - look at the second answer:<br /><br />http://splunk-base.splunk.com/answers/13896/scrub-data-from-splunk<br /><br />Also, check out this article in the community wiki:<br /><br />http://www.splunk.com/wiki/Community:Modifying_indexed_data_via_export_and_import

Sgoodman
August 12, 2011

Sfmike,<br /><br />When you clean an index, you need to precede the index name with the "-index" flag. (On the other hand, if you're cleaning all indexes, you don't use that flag, because you're not specifying any particular index.) Thanks for noticing the mistake in our examples; I've fixed that now.

Sgoodman
August 12, 2011

Thanson,<br /><br />Probably the easiest way to accomplish what you want is by using the "delete" search operator, as described early in this topic. Do a search that returns all data from the host. Once you've ascertained that you're getting the right search results, rerun the search command and pipe it through delete.

Sgoodman
August 12, 2011

what is the easiest way to purge data from a host that you are not longer monitoring? We decominsioned a server for example. Thanks

Thanson
August 11, 2011

clean examples do not match syntax description. It's unclear whether the user needs splunk clean eventdata -index or splunk clean eventdata

Sfmike
June 29, 2011

The delete operation marks matching index items to be excluded from future searches. So why isn't there something like "splunk clean expunge" that deletes items from the index that are marked as "deteled"?<br /><br />I do NOT want to run "splunk clean eventdata" as this article recommends because that would delete my entire index (which mostly includes data that cannot be reindexed since it came from a udp port 514 syslog stream).<br /><br />Or do I misunderstand how the syslog indexing works? I assumed it doesn't save the data anywhere except in the index.

Jnilsson
May 9, 2011

Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters