Create custom indexes

You can create two types of indexes:

• Events indexes
• Metrics indexes

Events indexes are the default index type. To create events indexes, see Create events indexes.

To create metrics indexes, see Create metrics indexes. For general information on metrics indexes, see the Metrics manual, starting with Overview of metrics.

For information on creating SmartStore indexes, see Add a SmartStore index.

Create events indexes

The main index, by default, holds all your events. It also serves as the default index for any inputs or search commands that don't specify an index, although you can change the default.

With a Splunk Enterprise license, you can add an unlimited number of additional indexes. You can add indexes using Splunk Web, the CLI, or indexes.conf.

This topic covers:

• The reasons why you might want multiple indexes.
• How to create new indexes.
• How to send events to specific indexes.
• How to search specific indexes.

Why have multiple indexes?

There are several key reasons for having multiple indexes:

• To control user access.
• To accommodate varying retention policies.
• To speed searches in certain situations.

The main reason you'd set up multiple indexes is to control user access to the data that's in them. When you assign users to roles, you can limit user searches to specific indexes based on the role they're in.

In addition, if you have different policies for retention for different sets of data, you might want to send the data to different indexes and then set a different archive or retention policy for each index.

Another reason to set up multiple indexes has to do with the way search works. If you have both a high-volume/high-noise data source and a low-volume data source feeding into the same index, and you search mostly for events from the low-volume data source, the search speed will be slower than necessary, because the indexer also has to search through all the data from the high-volume source. To mitigate this, you can create dedicated indexes for each data source and send data from each source to its dedicated index. Then, you can specify which index to search on. You'll probably notice an increase in search speed.

Create events indexes

You can create events indexes with Splunk Web, the CLI, or by editing indexes.conf directly.

Note: To add a new index to an indexer cluster, you must directly edit indexes.conf. You cannot add an index via Splunk Web or the CLI. For information on how to configure indexes.conf for clusters, see Configure the peer indexes in an indexer cluster. That topic includes an example of creating a new cluster index.

Use Splunk Web

You cannot use Splunk Web to add a SmartStore index. You also cannot use Splunk Web to add a non-SmartStore index, if the indexer has any SmartStore indexes.

1. In Splunk Web, navigate to Settings > Indexes and click New.
2. To create a new index, enter:
   • A name for the index. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen, or contain the word "kvstore".
   • The index data type. For event data, click Events. This is the default data type.
   • The path locations for index data storage:
     • Home path. Leave blank for default $SPLUNK_DB/<index_name>/db
     • Cold path. Leave blank for default $SPLUNK_DB/<index_name>/colddb
     • Thawed path. Leave blank for default $SPLUNK_DB/<index_name>/thaweddb
   • Enable/disable data integrity check.
   • The maximum size of the entire index. Defaults to 500000MB.
   • The maximum size of each index bucket. When setting the maximum size, use auto_high_volume for high volume indexes (such as the main index); otherwise, use auto.
   • The frozen archive path. Set this field if you want to archive frozen buckets. For information on bucket archiving, see Archive indexed data.
   • The app in which the index resides.
   • The tsidx retention policy. See Reduce tsidx usage.
   For more information on index settings, see Configure index storage.
3. Click Save.

You can edit an index by clicking on the index name in the Indexes section of the Settings menu in Splunk Web. Properties that you cannot change in Splunk Web are grayed out. To change those properties, edit indexes.conf, then restart the indexer.

Note: Some index properties are configurable only by editing the indexes.conf file. Check the indexes.conf topic for a complete list of properties.

Use the CLI

Navigate to the $SPLUNK_HOME/bin/ directory and use the add index command. You do not need to stop the indexer first.

To add a new index called "fflanda", enter the following command:

splunk add index fflanda

Note: User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen, or contain the word "kvstore".

If you do not want to use the default path for your new index, you can use parameters to specify a new location:

splunk add index foo -homePath /your/path/foo/db -coldPath /your/path/foo/colddb
    -thawedPath /your/path/foo/thawedDb

You can also edit an index's properties from the CLI. For example, to edit an index called "fflanda" using the CLI, type:

splunk edit index fflanda -<parameter> <value>

For detailed information on index settings, see Configure index storage.

Edit indexes.conf

To add a new index, add a stanza to indexes.conf in $SPLUNK_HOME/etc/system/local, identified by the name of the new index. For example:

[newindex]
homePath=<path for hot and warm buckets>
coldPath=<path for cold buckets>
thawedPath=<path for thawed buckets>
...

For information on index settings, see Configure index storage and the indexes.conf spec file.

Note: User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen, or contain the word "kvstore".

You must restart the indexer after editing indexes.conf.

For information on adding or editing index configurations on cluster nodes, see Configure the peer indexes in an indexer cluster.

Send events to specific indexes

By default, all external events go to the index called main. However, you might want to send some events to other indexes. For example, you might want to route all data from a particular input to its own index. Or you might want to segment data or send event data from a noisy source to an index that is dedicated to receiving it.

Important: To send events to a specific index, the index must already exist on the indexer. If you route any events to an index that doesn't exist, the indexer will drop those events.

Send all events from a data input to a specific index

To send all events from a particular data input to a specific index, add the following line to the input's stanza in inputs.conf on the Splunk Enterprise component where the data is entering the system: either the indexer itself or a forwarder sending data to the indexer:

index = <index_name>

The following example inputs.conf stanza sends all data from /var/log to an index named fflanda:

[monitor:///var/log]
disabled = false
index = fflanda

Route specific events to a different index

Just as you can route events to specific queues, you can also route specific events to specific indexes. You configure this on the indexer itself, not on the forwarder sending data to the indexer, if any.

To route certain events to a specific index, edit props.conf and transforms.conf on the indexer:

1. Identify a common attribute for the events that can be used to differentiate them.

2. In props.conf, create a stanza for the source, source type, or host. This stanza specifies a transforms_name that corresponds to a regex-containing stanza you will create in transforms.conf.

3. In transforms.conf, create an stanza named with the transforms_name you specified in step 2. This stanza:

• Specifies a regular expression that matches the identified attribute from step 1.
• Specifies the alternate index that events matching the attribute should be routed to.

The sections below fill out the details for steps 2 and 3.

Edit props.conf

Add the following stanza to $SPLUNK_HOME/etc/system/local/props.conf:

[<spec>]
TRANSFORMS-<class_name> = <transforms_name>

Note the following:

• <spec> is one of the following:
  • <sourcetype>, the sourcetype of an event
  • host::<host>, where <host> is the host for an event
  • source::<source>, where <source> is the source for an event

• <class_name> is any unique identifier.

• <transforms_name> is whatever unique identifier you want to give to your transform in transforms.conf.

Edit transforms.conf

Add the following stanza to $SPLUNK_HOME/etc/system/local/transforms.conf:

[<transforms_name>]
REGEX = <your_custom_regex>
DEST_KEY = _MetaData:Index
FORMAT = <alternate_index_name>

Note the following:

• <transforms_name> must match the <transforms_name> identifier you specified in props.conf.

• <your_custom_regex> must provide a match for the attribute you identified earlier, in step 1.

• DEST_KEY must be set to the index attribute _MetaData:Index.

• <alternate_index_name> specifies the alternate index that the events will route to.

Example

This examples routes events of windows_snare_log source type to the appropriate index based on their log types. "Application" logs will go to an alternate index, while all other log types, such as "Security", will go to the default index.

To make this determination, it uses props.conf to direct events of windows_snare_log source type through the transforms.conf stanza named "AppRedirect", where a regex then looks for the log type, "Application". Any event with a match on "Application" in the appropriate location is routed to the alternate index, "applogindex". All other events go to the default index.

1. Identify an attribute

The events in this example look like this:

web1.example.com	MSWinEventLog	1	Application	721	Wed Sep 06 17:05:31 2006
4156	MSDTC	Unknown User	N/A	Information	WEB1	Printers		String
message: Session idle timeout over, tearing down the session.	179

web1.example.com	MSWinEventLog	1	Security	722	Wed Sep 06 17:59:08 2006
576	Security	SYSTEM	User	Success Audit	WEB1	Privilege Use
Special privileges assigned to new logon:     User Name:      Domain:      Logon
ID: (0x0,0x4F3C5880)     Assigned: SeBackupPrivilege   SeRestorePrivilege
SeDebugPrivilege   SeChangeNotifyPrivilege   SeAssignPrimaryTokenPrivilege 525

Some events contain the value "Application", while others contain the value "Security" in the same location.

2. Edit props.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/props.conf:

[windows_snare_syslog]
TRANSFORMS-index = AppRedirect

This directs events of windows_snare_syslog sourcetype to the AppRedirect stanza in transforms.conf.

3. Edit transforms.conf

Add this stanza to $SPLUNK_HOME/etc/system/local/transforms.conf:

[AppRedirect]
REGEX = MSWinEventLog\s+\d+\s+Application
DEST_KEY = _MetaData:Index
FORMAT = applogindex

This stanza processes the events directed here by props.conf. Events that match the regex (because they contain the string "Application" in the specified location) get routed to the alternate index, "applogindex". All other events route as usual to the default index.

Search a specific index

When the indexer searches, it targets the default index (by default, main), unless the search explicitly specifies an index. For example, this search command searches in the hatch index:

index=hatch userid=henry.gale

You can also specify an alternate default index for a given role to search when you create or edit that role.

Create metrics indexes

You can create metrics indexes with Splunk Web, the CLI, the REST API, or by editing indexes.conf directly. For more about metrics, see Overview of metrics in the Metrics manual.

Use Splunk Web

1. In Splunk Web, navigate to Settings > Indexes and click New.
2. For Index Name, type a name for the index. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. Index names cannot begin with an underscore or hyphen, or contain the word "kvstore".
3. For Index Data Type, click Metrics.
4. (Optional) Set Timestamp Resolution to Milliseconds if you want the metrics index to store metric data points at that increased level of granularity. Metrics indexes with millisecond timestamp resolution have decreased search performance. See Metrics indexes with millisecond timestamps.
5. Enter the remaining properties of the index as needed. For details, see Create events indexes.
6. Click Save.

Use the command line interface (CLI)

1. Open a command prompt.
2. Navigate to the $SPLUNK_HOME/bin/ directory.
3. Use the add index command to create an index. User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. Index names cannot begin with an underscore or hyphen, or contain the word "kvstore".

For example, to create an index called mymetricsindex, enter the following command:

splunk add index mymetricsindex -datatype metric

To list all metrics indexes, enter the following command:

splunk list index -datatype metric

To list all indexes, including events indexes, enter the following command:

splunk list index -datatype all

Use the REST API

Create an index using the /data/indexes endpoint with the "datatype=metric" parameter. For details, see /data/indexes in the REST API Reference Manual.

For example, to create a metrics index called mymetricsindex, enter the following command:

curl -k -u admin:pass https://localhost:8089/services/data/indexes  \
    -d name=mymetricsindex                                          \
    -d datatype=metric

To list all metrics indexes using the REST API, enter the following command:

curl -k -u admin:pass https://localhost:8089/services/data/indexes?datatype=metric

To list all indexes, including events indexes, enter the following command:

curl -k -u admin:pass https://localhost:8089/services/data/indexes?datatype=all

Edit indexes.conf

To create a new metrics index, add a stanza to indexes.conf in $SPLUNK_HOME/etc/system/local, identified by the name of the new index. Change the datatype parameter to datatype = metric.

For example, to create a metrics index called "mymetricsindex", add the following stanza:

[mymetricsindex]
homePath=<path for hot and warm buckets>
coldPath=<path for cold buckets>
thawedPath=<path for thawed buckets>
datatype = metric
metric.timestampResolution = <s | ms>
...

For information on index settings, see Configure index storage and the indexes.conf spec file.

User-defined index names must consist of only numbers, lowercase letters, underscores, and hyphens. They cannot begin with an underscore or hyphen, or contain the word "kvstore".

You must restart the indexer after editing indexes.conf.

For information on adding or editing index configurations on cluster nodes, see Configure the peer indexes in an indexer cluster.

Metrics indexes with millisecond timestamps

By default, metrics indexes are only searchable at a second-by-second precision. This is unlike events indexes, which can be searched with subsecond precision by default.

If you are dealing with a high volume source of metric data, such as a utility grid that has the potential to generate millions of metric data points per second, this means that the metric index is populated with sample metric data points or metric data points that are aggregated views of the raw metric data, taken at regular intervals.

If you are concerned about high index volume, this can be a good thing. Having second precision metrics indexes keeps your indexes lean and saves you from having to search through huge numbers of events over relatively short time ranges. But this also means that you cannot run time-based metrics searches that have subsecond precision. Similarly, you cannot set up mstats searches that group by subsecond span values.

If you need the capability to perform metric searches with subsecond precision, give your new metric index a Timestamp Resolution of Milliseconds. Metrics indexes with millisecond timestamp resolution can have decreased search performance in comparison to metrics indexes that have the default second timestamp precision.

Metrics indexes set to millisecond precision might incur more license usage than similar metrics indexes set to second precision. The license cost per metric data point remains the same, but millisecond-precision indexes can index more data points than second-precision indexes ingesting data from the same source.

About changing timestamp resolutions of metrics indexes

You can change the timestamp resolution of a metrics index after you create it. However, if you change the timestamp resolution of a metrics index from millisecond to second, it may look like data loss to people who regularly run searches against that metrics index. This is because the index won't ingest data at millisecond resolution after the change.

When your index is at millisecond timestamp resolution, your indexed metric data points might have timestamps like this.

_timestamp (seconds)
1.000
1.001
1.002
2.000
2.435
3.123
3.651
4.000

After four seconds, if you change the timestamp resolution from millisecond timestamp resolution to second timestamp resolution, your index is restricted to indexing one metric data point per second:

_timestamp (seconds)
5.000
6.000
7.000
8.000
9.000

Some users may perceive this as a data loss when in fact they are just seeing their data with a less granular timestamp resolution.

Similarly, users of a metrics index that is switched from a second timestamp resolution to a millisecond timestamp resolution may be surprised to see their indexes ingesting more events than they did before the switch.

As an administrator of a Splunk Enterprise deployment it is up to you to communicate this change and its implications to your users.