Documentation > Splunk > Admin Manual > Set up multiple indexes

Admin Manual

 


Welcome to Splunk administration
What to do first
Use Splunk Web
Getting started for Windows admins
Manage Splunk licenses
Meet Splunk apps
Configure Splunk
Indexing with Splunk
Manage users
Manage indexes
Define alerts
Set up backups and retention policies
Configure data security
Manage search jobs
Use Splunk's command line interface (CLI)
Configuration file reference

Set up multiple indexes

Contents

Set up multiple indexes

Splunk ships with an index called main that, by default, holds all your events. By default, Splunk also creates a number of other indexes for use by its internal systems, as well as for additional Splunk features such as summary indexing and event auditing.

Splunk with an Enterprise license lets you add an unlimited number of additional indexes. The main index serves as the default index for any input or search command that doesn't specify an index, although you can change the default. You can add indexes using Splunk Web, Splunk's CLI, or indexes.conf.

Why have multiple indexes?

There are several key reasons for having multiple indexes:

The main reason you'd set up multiple indexes is to control user access to the data that's in them. When you assign users to roles, you can limit user searches to specific indexes based on the role they're in.

In addition, if you have different policies for retention for different sets of data, you might want to send the data to different indexes and then set a different archive or retention policy for each index.

Another reason to set up multiple indexes has to do with the way Splunk search works. If you have both a high-volume/high-noise data source and a low-volume data source feeding into the same index, and you search mostly for events from the low-volume data source, the search speed will be slower than necessary, because Splunk also has to search through all the data from the high-volume source. To mitigate this, you can create dedicated indexes for each data source and route data from each source to its dedicated index. Then, you can specify which index to search on. You'll probably notice an increase in search speed.

Specify an index or indexes to search

When Splunk searches, it targets the default index (by default, main) unless otherwise specified. If you have created a new index, or want to search in any index that is not default, you can specify the index in your search:

index=hatch userid=henry.gale

This searches in the hatch index for the userid=henry.gale.

You can also specify an alternate default index for a given role to search when you create or edit that role.

Create and edit indexes

You can create or edit indexes with Splunk Web, the Splunk CLI, or directly, via indexes.conf.

Use Splunk Web

1. In Splunk Web, navigate to Manager > Indexes and click New.

2. To create a new index, enter:

3. When you've set the values you want, click Save. The index is created.

4. You must restart Splunk when you create a new index.

You can edit an index by clicking on the index name in the Indexes section of Manager in Splunk Web. Properties that you cannot change are grayed out. To change these properties, use indexes.conf. You must restart Splunk if you edit the properties of an existing index.

Note: Some index properties are configurable only when you create or edit indexes with the indexes.conf file. Check the indexes.conf topic for a complete list of properties.

Use the CLI

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command.

Important: You must stop Splunk before you edit the properties of an existing index. You do not stop Splunk before you create a new index.

To add or edit a new index called "fflanda" using the CLI: 

./splunk [add|edit] index fflanda

Note: Index names must consist of only numbers, letters, periods, underscores, and hyphens. They cannot begin with an underscore or hyphen.

If you do not want to use the default path for your new index, you can use parameters to specify a new location: 

./splunk add index foo -homePath /your/path/foo/db -coldPath /your/path/foo/colddb -thawedPath /your/path/foo/thawedDb

You must restart Splunk after you create a new index or edit the properties of an existing index.

Edit indexes.conf

To add a new index, add a stanza to indexes.conf in $SPLUNK_HOME/etc/system/local, identified by the name of the new index. See configuration details and examples in the indexes.conf topic.

You must restart Splunk to enable configuration changes.

Important: When creating a new index, the maximum number of hot buckets now defaults to 3. (Prior to version 4.2, the default was 1.) To change this, set the maxHotBuckets attribute in the new index's stanza in indexes.conf.

Disable an index

You can disable an index in Splunk Web. To do this, navigate to Manager > Indexes and click Disable to the right of the index you want to disable.

To delete an index or the data in the index, see "Remove indexed data from Splunk" in this manual.

Route events to specific indexes

Just as you can route events to specific queues, you can also route events to specific indexes.

By default, Splunk sends all events to the index called main. However, you may want to send specific events to other indexes. For example, you might want to segment data or to send event data from a noisy source to an index that is dedicated to receiving it. You can route data locally or route data you are receiving from remote sources or Splunk instances.

Note: When you place data in an alternate index, you must specify the index in your search with the index= command when you want to search that index:

index=foo

Send all events from a data input to a specific index

To configure routing for all events from a particular data input to an alternate index, add the following to the appropriate stanza in inputs.conf.

index = myindex

The following example inputs.conf entry routes data to index = fflanda: 

[monitor:///var/log]
disabled = false
index = fflanda

If you specify a different index on a forwarder, when the events reach the indexing instance they will be routed to the named index, which must already exist.

Route specific events to a different index

To route certain events to an alternate index, edit props.conf and transforms.conf on the local Splunk instance:

1. Identify a common attribute for the events that can be used to differentiate them.

2. In props.conf, create a stanza for the source, source type, or host. This stanza specifies a transforms_name that corresponds to a regex-containing stanza you will create in transforms.conf.

3. In transforms.conf, create an stanza named with the transforms_name you specified in step 2. This stanza:

The sections below fill out the details for steps 2 and 3.

Edit props.conf

Add the following stanza to $SPLUNK_HOME/etc/system/local/props.conf: 

[<spec>]
TRANSFORMS-<class_name> = <transforms_name>

Note the following:

Edit transforms.conf

Add the following stanza to $SPLUNK_HOME/etc/system/local/transforms.conf: 

[<transforms_name>]
REGEX = <your_custom_regex>
DEST_KEY = _MetaData:Index
FORMAT = <alternate_index_name>

Note the following:

Example

In this example, we route events of windows_snare_log sourcetype to the appropriate index based on their log types. "Application" logs will go to an alternate index, while all other log types, such as "Security", will go to the default index.

To make this determination, we use props.conf to direct events of windows_snare_log sourcetype through the transforms.conf stanza named "AppRedirect", where a regex then looks for the log type, "Application". Any event with a match on "Application" in the appropriate location is routed to the alternate index, "applogindex". All other events go to the default index.

Identify an attribute

The events in this example look like this: 

web1.example.com	MSWinEventLog	1	Application	721	Wed Sep 06 17:05:31 2006
4156	MSDTC	Unknown User	N/A	Information	WEB1	Printers		String
message: Session idle timeout over, tearing down the session.	179

web1.example.com	MSWinEventLog	1	Security	722	Wed Sep 06 17:59:08 2006
576	Security	SYSTEM	User	Success Audit	WEB1	Privilege Use
Special privileges assigned to new logon:     User Name:      Domain:      Logon
ID: (0x0,0x4F3C5880)     Assigned: SeBackupPrivilege   SeRestorePrivilege
SeDebugPrivilege   SeChangeNotifyPrivilege   SeAssignPrimaryTokenPrivilege 525

Some events contain the value "Application", while others contain the value "Security" in the same location.

Edit props.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/props.conf: 

[windows_snare_syslog]
TRANSFORMS-index = AppRedirect

This directs events of windows_snare_syslog sourcetype to the AppRedirect stanza in transforms.conf.

Edit transforms.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/transforms.conf: 

[AppRedirect]
REGEX = MSWinEventLog\s+\d+\s+Application
DEST_KEY = _MetaData:Index
FORMAT = applogindex

This stanza processes the events directed here by props.conf. Events that match the regex, by containing the string "Application" in the specified location, get routed to the alternate index, "applogindex". All other events route to the default index.

Retrieved from "http://docs.splunk.com/Documentation/Splunk/4.3/Admin/Setupmultipleindexes"

This documentation applies to the following versions of Splunk: 4.2 , 4.2.1 , 4.2.2 , 4.2.3 , 4.2.4 , 4.2.5 , 4.3 , 4.3.1 , 4.3.2 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!