Set up multiple indexes
Splunk ships with an index called
main that, by default, holds all your events. Splunk also creates a number of other indexes for use by its internal systems, as well as for additional Splunk features such as summary indexing and event auditing.
Splunk with an Enterprise license lets you add an unlimited number of additional indexes. The
main index serves as the default index for any input or search command that doesn't specify an index, although you can change the default. You can add indexes using Splunk Web, Splunk's CLI, or indexes.conf.
This topic covers:
- The reasons why you might want multiple indexes.
- How to create new indexes.
- How to send events to specific indexes.
- How to search specific indexes.
Why have multiple indexes?
There are several key reasons for having multiple indexes:
- To control user access.
- To accommodate varying retention policies.
- To speed searches in certain situations.
The main reason you'd set up multiple indexes is to control user access to the data that's in them. When you assign users to roles, you can limit user searches to specific indexes based on the role they're in.
In addition, if you have different policies for retention for different sets of data, you might want to send the data to different indexes and then set a different archive or retention policy for each index.
Another reason to set up multiple indexes has to do with the way Splunk search works. If you have both a high-volume/high-noise data source and a low-volume data source feeding into the same index, and you search mostly for events from the low-volume data source, the search speed will be slower than necessary, because Splunk also has to search through all the data from the high-volume source. To mitigate this, you can create dedicated indexes for each data source and send data from each source to its dedicated index. Then, you can specify which index to search on. You'll probably notice an increase in search speed.
Create and edit indexes
You can create or edit indexes with Splunk Web, the Splunk CLI, or by editing
Note: To add a new index to a cluster, you must directly edit
indexes.conf. You cannot add an index via Splunk Web or the CLI. For information on how to configure
indexes.conf for clusters, see "Configure the peer indexes". That topic includes an example of creating a new cluster index.
Use Splunk Web
1. In Splunk Web, navigate to Manager > Indexes and click New.
2. To create a new index, enter:
- A name for the index. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen.
- The path locations for index data storage:
- Home path; leave blank for default
- Cold db path; leave blank for default
- Thawed/resurrected db path, leave blank for default
- Home path; leave blank for default
- The maximum size of the entire index. Defaults to 500000MB.
- The maximum size of the hot (currently written to) portion of this index. When setting the maximum size, you should use
auto_high_volumefor high volume indexes (such as the main index); otherwise, use
- The frozen archive path. Set this field if you want Splunk to archive frozen buckets. For information on bucket archiving, see "Archive indexed data".
Note: For detailed information on each of these settings, see "Configure index storage".
3. Click Save.
You can edit an index by clicking on the index name in the Indexes section of the Manager menu in Splunk Web. Properties that you cannot change in Splunk Web are grayed out. To change those properties, edit
indexes.conf, then restart Splunk.
Note: Some index properties are configurable only by editing the
indexes.conf file. Check the
indexes.conf topic for a complete list of properties.
Use the CLI
Navigate to the
$SPLUNK_HOME/bin/ directory and use the
add index command. You do not need to stop Splunk first.
To add a new index called "fflanda", enter the following command:
splunk add index fflanda
Note: User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen.
If you do not want to use the default path for your new index, you can use parameters to specify a new location:
./splunk add index foo -homePath /your/path/foo/db -coldPath /your/pat/foo/colddb -thawedPath /your/path/foo/thawedDb
You can also edit an index's properties from the CLI. For example, to edit an index called "fflanda" using the CLI, type:
splunk edit index fflanda -<parameter> <value>
For detailed information on index settings, see "Configure index storage".
To add a new index, add a stanza to
$SPLUNK_HOME/etc/system/local, identified by the name of the new index. For example:
[newindex] homePath=<path for hot and warm buckets> coldPath=<path for cold buckets> thawedPath=<path for thawed buckets> ...
Note: The index name, specified in the stanza, must consist of only numbers, lowercase letters, underscores, and hyphens. User-defined index names cannot begin with an underscore or hyphen.
You must restart Splunk after editing
Important: For information on adding or editing index configurations on cluster nodes, see "Configure the peer indexes".
Send events to specific indexes
By default, Splunk sends all events to the index called main. However, you might want to send some events to other indexes. For example, you might want to route all data from a particular input to its own index. Or you might want to segment data or send event data from a noisy source to an index that is dedicated to receiving it.
Important: To send events to a specific index, the index must already exist on the indexer. If you route any events to an index that doesn't exist, the indexer will drop those events.
Send all events from a data input to a specific index
To send all events from a particular data input to a specific index, add the following line to the input's stanza in inputs.conf on the Splunk component where the data is entering the system: either the indexer itself or a forwarder sending data to the indexer:
index = <index_name>
The following example
inputs.conf stanza sends all data from
/var/log to an index named
[monitor:///var/log] disabled = false index = fflanda
Route specific events to a different index
Just as you can route events to specific queues, you can also route specific events to specific indexes. You configure this on the indexer itself, not on the forwarder sending data to the indexer, if any.
1. Identify a common attribute for the events that can be used to differentiate them.
props.conf, create a stanza for the source, source type, or host. This stanza specifies a
transforms_name that corresponds to a regex-containing stanza you will create in
transforms.conf, create an stanza named with the
transforms_name you specified in step 2. This stanza:
- Specifies a regular expression that matches the identified attribute from step 1.
- Specifies the alternate index that events matching the attribute should be routed to.
The sections below fill out the details for steps 2 and 3.
Add the following stanza to
[<spec>] TRANSFORMS-<class_name> = <transforms_name>
Note the following:
<spec>is one of the following:
<sourcetype>, the sourcetype of an event
<host>is the host for an event
<source>is the source for an event
<class_name>is any unique identifier.
<transforms_name>is whatever unique identifier you want to give to your transform in
Add the following stanza to
[<transforms_name>] REGEX = <your_custom_regex> DEST_KEY = _MetaData:Index FORMAT = <alternate_index_name>
Note the following:
<transforms_name>must match the
<transforms_name>identifier you specified in
<your_custom_regex>must provide a match for the attribute you identified earlier, in step 1.
DEST_KEYmust be set to the index attribute
<alternate_index_name>specifies the alternate index that the events will route to.
This examples routes events of
windows_snare_log source type to the appropriate index based on their log types. "Application" logs will go to an alternate index, while all other log types, such as "Security", will go to the default index.
To make this determination, it uses
props.conf to direct events of
windows_snare_log source type through the
transforms.conf stanza named "AppRedirect", where a regex then looks for the log type, "Application". Any event with a match on "Application" in the appropriate location is routed to the alternate index, "applogindex". All other events go to the default index.
1. Identify an attribute
The events in this example look like this:
web1.example.com MSWinEventLog 1 Application 721 Wed Sep 06 17:05:31 2006 4156 MSDTC Unknown User N/A Information WEB1 Printers String message: Session idle timeout over, tearing down the session. 179 web1.example.com MSWinEventLog 1 Security 722 Wed Sep 06 17:59:08 2006 576 Security SYSTEM User Success Audit WEB1 Privilege Use Special privileges assigned to new logon: User Name: Domain: Logon ID: (0x0,0x4F3C5880) Assigned: SeBackupPrivilege SeRestorePrivilege SeDebugPrivilege SeChangeNotifyPrivilege SeAssignPrimaryTokenPrivilege 525
Some events contain the value "Application", while others contain the value "Security" in the same location.
2. Edit props.conf
Add this stanza to
[windows_snare_syslog] TRANSFORMS-index = AppRedirect
This directs events of
windows_snare_syslog sourcetype to the
AppRedirect stanza in
3. Edit transforms.conf
Add this stanza to
[AppRedirect] REGEX = MSWinEventLog\s+\d+\s+Application DEST_KEY = _MetaData:Index FORMAT = applogindex
This stanza processes the events directed here by
props.conf. Events that match the regex (because they contain the string "Application" in the specified location) get routed to the alternate index, "applogindex". All other events route as usual to the default index.
Search a specific index
When Splunk searches, it targets the default index (by default, main), unless the search explicitly specifies an index. For example, this search command searches in the
You can also specify an alternate default index for a given role to search when you create or edit that role.
About managing indexes
Remove indexes and data from Splunk
This documentation applies to the following versions of Splunk® Enterprise: 5.0, 5.0.1, 5.0.2, 5.0.3, 5.0.4, 5.0.5, 5.0.6, 5.0.7, 5.0.8, 5.0.9, 5.0.10, 5.0.11, 5.0.12, 5.0.13, 5.0.14, 5.0.15, 5.0.16, 5.0.17, 5.0.18