Documentation > Splunk > REST API Reference > Indexes

REST API Reference

 


REST API reference
Using Splunk's REST API

Indexes

Contents

Indexes

Use the Indexes endpoints to create and manage data indexes.

data/indexes

Provides services to create and manage data indexes.

GET data/indexes

Lists the recognized indexes on the server.

Request

Name Type Required Default Description
count Number 30 Indicates the maximum number of entries to return. To return all entries, specify -1.
offset Number 0 Index for first item to return.
search String Search expression to filter the response. The response matches field values against the search expression. For example:

search=foo matches any object that has "foo" as a substring in a field. search=field_name%3Dfield_value restricts the match to a single field. URI-encoding is required in this example.

sort_dir Enum asc Valid values: (asc | desc)

Indicates whether to sort returned entries in ascending or descending order.

sort_key String name Field to use for sorting.
sort_mode Enum auto Valid values: (auto | alpha | alpha_case | num)

Indicates the collating sequence for sorting the returned entries. auto: If all values of the field are numbers, collate numerically. Otherwise, collate alphabetically. alpha: Collate alphabetically. alpha_case: Collate alphabetically, case-sensitive. num: Collate numerically.

summarize Bool If true, leaves out certain index details in order to provide a faster response.

Response Codes

Status Code Description
200 OK
400 Request error. See response body for details.
401 Authentication failure: must pass valid credentials with request.
403 Insufficient permissions to view indexes.
409 Request error: this operation is invalid for this item. See response body for details.
500 Internal server error. See response body for details.

Returned Values

Attribute Description
assureUTF8 Indicates whether all data retreived from the index is proper UTF8. If enabled (set to True), degrades indexing performance.

This is a global setting, not a per index setting.

blockSignSize Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

blockSignatureDatabase The index that stores block signatures of events.

This is a global setting, not a per index setting.

coldPath Filepath to the cold databases for the index.
coldPath_expanded Absoute filepath to the cold databases.
coldToFrozenDir Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

  • New style buckets (4.2 and on): removes all files but the rawdata
To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory
  • Old style buckets (Pre-4.2): gzip all the .data and .tsidx files
To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence.

coldToFrozenScript Path to the archiving script.

See the POST parameter description for details.

compressRawdata This value is ignored. splunkd process always compresses raw data.
currentDBSizeMB Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.
defaultDatabase If no index destination information is available in the input data, the index shown here is the destination of such data.
disabled Indicates if the index is disabled.
enableRealtimeSearch Indicates if this is a real-time search.

This is a global setting, not a per index setting.

frozenTimePeriodInSecs Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath An absolute path that contains the hot and warm buckets for the index.
homePath_expanded An absolute filepath to the hot and warm buckets for the index.
indexThreads Number of threads used for indexing.

This is a global setting, not a per index setting.

isInternal Indicates if this is an internal index (for example, _internal, _audit).
lastInitTime Last time the index processor was successfully initialized.

This is a global setting, not a per index setting.

maxConcurrentOptimizes The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk to autotune this parameter (recommended). Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" is typically one that gets over 10GB of data per day.
  • "auto" sets the size to 750MB.
  • "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize will be auto tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

If a hot bucket exceeds maxHotIdleSecs, Splunk rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.

maxHotSpanSecs Upper bound of target maximum timespan of hot/warm buckets in seconds. Defaults to 7776000 seconds (90 days).

Note: If set too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB The amount of memory, in MB, allocated for indexing.

This is a global setting, not a per index setting.

maxMetaEntries Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you don't use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxRunningProcessGroups Maximum number of processes that the indexer fires off at a time.

This is a global setting, not a per index setting.

maxTime UNIX timestamp of the newest event time in the index.
maxTotalDataSizeMB The maximum size of an index, in MB.
maxWarmDBCount The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.
memPoolMB Determines how much memory is given to the indexer memory pool.

This is a global setting, not a per-index setting.

minRawFileSyncSecs Can be either an integer (or "disable"). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

The integer sets how frequently splunkd forces a filesystem sync while compressing journal slices.

During this interval, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

minTime UNIX timestamp of the oldest event time in the index.
partialServiceMetaPeriod Related to serviceMetaPeriod. By default it is turned off (zero).

If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

quarantineFutureSecs Events with timestamp of quarantineFutureSecs newer than "now" that are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

Warning: This is an advanced parameter. Only change it if instructed to do so by Splunk Support.

rotatePeriodInSecs Rotation period, in seconds, that specifies how frequently to check:
  • If a new hot bucket needs to be created.
  • If there are any cold buckets that should be frozen.
  • If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.
serviceMetaPeriod Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

summarize If true, leaves out certain index details, which provides a faster response.
suppressBannerList List of indexes for which we suppress "index missing" warning banner messages.

This is a global setting, not a per index setting.

sync Specifies the number of events that trigger the indexer to sync events.

This is a global setting, not a per index setting.

syncMeta When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of Splunk Support.

thawedPath An absolute path that contains the thawed (resurrected) databases for the index.
thawedPath_expanded Absolute filepath to the thawed (resurrected) databases.
throttleCheckPeriod Defines how frequently Splunk checks for index throttling condition, in seconds. Defaults to 15 (seconds).

Note: Do not change this parameter without the input of Splunk Support.

totalEventCount Total number of events in the index.

Example

Lists the indexes on this Splunk instance. 


curl -k -u admin:pass https://localhost:8089/services/data/indexes



<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"
      xmlns:s="http://dev.splunk.com/ns/rest">
  <title>indexes</title>
  <id>https://localhost:8089/services/data/indexes</id>
  <updated>2011-07-11T18:09:22-07:00</updated>
  <generator version="102807"/>
  <author>
    <name>Splunk</name>
  </author>
  <link href="/services/data/indexes/_new" rel="create"/>
  <link href="/services/data/indexes/_reload" rel="_reload"/>
  <!-- opensearch nodes elided for brevity. -->
  <s:messages/>
  <entry>
    <title>_audit</title>
    <id>https://localhost:8089/servicesNS/nobody/system/data/indexes/_audit</id>
    <updated>2011-07-11T18:09:22-07:00</updated>
    <link href="/servicesNS/nobody/system/data/indexes/_audit" rel="alternate"/>
    <author>
      <name>nobody</name>
    </author>
    <link href="/servicesNS/nobody/system/data/indexes/_audit" rel="list"/>
    <link href="/servicesNS/nobody/system/data/indexes/_audit/_reload" rel="_reload"/>
    <link href="/servicesNS/nobody/system/data/indexes/_audit" rel="edit"/>
    <link href="/servicesNS/nobody/system/data/indexes/_audit/disable" rel="disable"/>
    <content type="text/xml">
      <s:dict>
        <s:key name="assureUTF8">0</s:key>
        <s:key name="blockSignSize">0</s:key>
        <s:key name="blockSignatureDatabase">_blocksignature</s:key>
        <s:key name="coldPath">$SPLUNK_DB/audit/colddb</s:key>
        <s:key name="coldPath_expanded">/home/amrit/temp/curl/splunk/var/lib/splunk/audit/colddb</s:key>
        <s:key name="coldToFrozenDir"/>
        <s:key name="coldToFrozenScript"/>
        <s:key name="compressRawdata">1</s:key>
        <s:key name="currentDBSizeMB">1</s:key>
        <s:key name="defaultDatabase">main</s:key>
        <s:key name="disabled">0</s:key>
        <!-- eai:acl nodes elided for brevity. -->
        <s:key name="enableRealtimeSearch">1</s:key>
        <s:key name="frozenTimePeriodInSecs">188697600</s:key>
        <s:key name="homePath">$SPLUNK_DB/audit/db</s:key>
        <s:key name="homePath_expanded">/home/amrit/temp/curl/splunk/var/lib/splunk/audit/db</s:key>
        <s:key name="indexThreads">auto</s:key>
        <s:key name="isInternal">1</s:key>
        <s:key name="lastInitTime">1310432962.424512</s:key>
        <s:key name="maxConcurrentOptimizes">3</s:key>
        <s:key name="maxDataSize">auto</s:key>
        <s:key name="maxHotBuckets">3</s:key>
        <s:key name="maxHotIdleSecs">0</s:key>
        <s:key name="maxHotSpanSecs">7776000</s:key>
        <s:key name="maxMemMB">5</s:key>
        <s:key name="maxMetaEntries">1000000</s:key>
        <s:key name="maxRunningProcessGroups">20</s:key>
        <s:key name="maxTime">2011-07-10T22:20:53-0700</s:key>
        <s:key name="maxTotalDataSizeMB">500000</s:key>
        <s:key name="maxWarmDBCount">300</s:key>
        <s:key name="memPoolMB">auto</s:key>
        <s:key name="minRawFileSyncSecs">disable</s:key>
        <s:key name="minTime">2011-07-10T14:33:00-0700</s:key>
        <s:key name="partialServiceMetaPeriod">0</s:key>
        <s:key name="quarantineFutureSecs">2592000</s:key>
        <s:key name="quarantinePastSecs">77760000</s:key>
        <s:key name="rawChunkSizeBytes">131072</s:key>
        <s:key name="rotatePeriodInSecs">60</s:key>
        <s:key name="serviceMetaPeriod">25</s:key>
        <s:key name="suppressBannerList"/>
        <s:key name="sync">0</s:key>
        <s:key name="syncMeta">1</s:key>
        <s:key name="thawedPath">$SPLUNK_DB/audit/thaweddb</s:key>
        <s:key name="thawedPath_expanded">/home/amrit/temp/curl/splunk/var/lib/splunk/audit/thaweddb</s:key>
        <s:key name="throttleCheckPeriod">15</s:key>
        <s:key name="totalEventCount">230</s:key>
      </s:dict>
    </content>
  </entry>
</feed>

POST data/indexes

Creates a new index with the given name.

Request

Name Type Required Default Description
name String
✓
 The name of the index to create.
blockSignSize Number 0 Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

bucketRebuildMemoryHint String auto Suggestion for the Splunk bucket rebuild process for the size of the time-series (tsidx) file to make.

Caution: This is an advanced parameter. Inappropriate use of this parameter causes splunkd to not start if rebuild is required. Do not set this parameter unless instructed by Splunk Support.

Default value, auto, varies by the amount of physical RAM on the host

  • less than 2GB RAM = 67108864 (64MB) tsidx
  • 2GB to 8GB RAM = 134217728 (128MB) tsidx
  • more than 8GB RAM = 268435456 (256MB) tsidx

Values other than "auto" must be 16MB-1GB. Highest legal value (of the numerical part) is 4294967295

You can specify the value using a size suffix: "16777216" or "16MB" are equivalent.

coldPath String An absolute path that contains the colddbs for the index. The path must be readable and writable. Cold databases are opened as needed when searching. May be defined in terms of a volume definition (see volume section below).

Required. Splunk will not start if an index lacks a valid coldPath.

coldToFrozenDir String Destination path for the frozen archive. Use as an alternative to a coldToFrozenScript. Splunk automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

  • New style buckets (4.2 and on): removes all files but the rawdata
To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory
  • Old style buckets (Pre-4.2): gzip all the .data and .tsidx files
To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence

coldToFrozenScript String Path to the archiving script.

If your script requires a program to run it (for example, python), specify the program followed by the path. The script must be in $SPLUNK_HOME/bin or one of its subdirectories.

Splunk ships with an example archiving script in $SPLUNK_HOME/bin called coldToFrozenExample.py. Splunk DOES NOT recommend using this example script directly. It uses a default path, and if modified in place any changes will be overwritten on upgrade.

Splunk recommends copying the example script to a new file in bin and modifying it for your system. Most importantly, change the default archive path to an existing directory that fits your needs.

If your new script in bin/ is named myColdToFrozen.py, set this key to the following:

coldToFrozenScript = "$SPLUNK_HOME/bin/python" "$SPLUNK_HOME/bin/myColdToFrozen.py"

By default, the example script has two possible behaviors when archiving:

  • For buckets created from version 4.2 and on, it removes all files except for rawdata. To thaw: cd to the frozen bucket and type splunk rebuild ., then copy the bucket to thawed for that index. We recommend using the coldToFrozenDir parameter unless you need to perform a more advanced operation upon freezing buckets.
  • For older-style buckets, we simply gzip all the .tsidx files. To thaw: cd to the frozen bucket and unzip the tsidx files, then copy the bucket to thawed for that index
compressRawdata Boolean true This parameter is ignored. The splunkd process always compresses raw data.
enableOnlineBucketRepair Boolean true Enables asynchronous "online fsck" bucket repair, which runs concurrently with Splunk.

When enabled, you do not have to wait until buckets are repaired to start Splunk. However, you might observe a slight performance degratation.

Note: This endpoint is new in Splunk 4.3.

frozenTimePeriodInSecs Number 188697600 Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath String An absolute path that contains the hot and warm buckets for the index.

Required. Splunk will not start if an index lacks a valid homePath.

CAUTION: Path MUST be readable and writable.

maxBloomBackfillBucketAge Number 30d Valid values are: Integer[m|s|h|d]

If a warm or cold bucket is older than the specified age, do not create or rebuild its bloomfilter. Specify 0 to never rebuild bloomfilters.

For example, if a bucket is older than specified with maxBloomBackfillBucketAge, and the rebuilding of its bloomfilter started but did not finish, do not rebuild it.

maxConcurrentOptimizes Number 3 The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize Number auto The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk to autotune this parameter (recommended).Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" would typically be considered one that gets over 10GB of data per day.
  • "auto" sets the size to 750MB.
  • "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize will be auto tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets Number 3 Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs Number 0 Maximum life, in seconds, of a hot bucket. Defaults to 0.

If a hot bucket exceeds maxHotIdleSecs, Splunk rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

maxHotSpanSecs Number 7776000 Upper bound of target maximum timespan of hot/warm buckets in seconds. Defaults to 7776000 seconds (90 days).

Note:I f you set this too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB Number 5 The amount of memory, expressed in MB, to allocate for buffering a single tsidx file into memory before flushing to disk. Defaults to 5. The default is recommended for all environments.

IMPORTANT: Calculate this number carefully. Setting this number incorrectly may have adverse effects on your systems memory and/or splunkd stability/performance.

maxMetaEntries Number 1000000 Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you don't use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxTimeUnreplicatedNoAcks Number 300 Upper limit, in seconds, on how long an event can sit in raw slice. Applies only if replication is enabled for this index. Otherwise ignored.

If there are any acknowledged events sharing this raw slice, this paramater does not apply. In this case, maxTimeUnreplicatedWithAcks applies.

Highest legal value is 2147483647. To disable this parameter, set to 0.

Note: this is an advanced parameter. Understand the consequences before changing.

maxTimeUnreplicatedWithAcks Number 60 Upper limit, in seconds, on how long events can sit unacknowledged in a raw slice. Applies only if you have enabled acks on forwarders and have replication enabled (with clustering).

Note: This is an advanced parameter. Make sure you understand the settings on all forwarders before changing this. This number should not exceed ack timeout configured on any forwarder, and should actually be set to at most half of the minimum value of that timeout. You can find this setting in outputs.conf readTimeout setting under the tcpout stanza.

To disable, set to 0, but this is NOT recommended. Highest legal value is 2147483647.

maxTotalDataSizeMB Number 500000 The maximum size of an index (in MB). If an index grows larger than the maximum size, the oldest data is frozen.
maxWarmDBCount Number 300 The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times will be moved to cold.
minRawFileSyncSecs Number disable Specify an integer (or "disable") for this parameter.

This parameter sets how frequently splunkd forces a filesystem sync while compressing journal slices.

During this interval, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

Note: Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

partialServiceMetaPeriod Number 0 Related to serviceMetaPeriod. If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

By default it is turned off (zero).

quarantineFutureSecs Number 2592000 Events with timestamp of quarantineFutureSecs newer than "now" are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs Number 77760000 Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes Number 131072 Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

WARNING: This is an advanced parameter. Only change it if you are instructed to do so by Splunk Support.

repFactor String 0 Value is either a non-negative number or "auto." This parameter only applies to Splunk clustering slaves.

auto: Use the value as configured with the master.

0: Specify zero to turn off replication for this index.

For information on configuring clusters, see Configure clusters in the Splunk Managing Indexing and Clusters manual.

rotatePeriodInSecs Number 60 How frequently (in seconds) to check if a new hot bucket needs to be created. Also, how frequently to check if there are any warm/cold buckets that should be rolled/frozen.
serviceMetaPeriod Number 25 Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

syncMeta Boolean true When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of a Splunk Support.

thawedPath String An absolute path that contains the thawed (resurrected) databases for the index.

Cannot be defined in terms of a volume definition.

Required. Splunk will not start if an index lacks a valid thawedPath</codePath>.


throttleCheckPeriod Number 15 Defines how frequently Splunk checks for index throttling condition, in seconds. Defaults to 15 (seconds).

Note: Do not change this parameter without the input of Splunk Support.

Response Codes

Status Code Description
201 Index created successfully; followed by header:

<code>Location: /services/data/indexes/{name}

400 Request error. See response body for details.
401 Authentication failure: must pass valid credentials with request.
402 The Splunk license in use has disabled this feature.
403 Insufficient permissions to create index.
409 The index name already exists.
500 Internal server error. See response body for details.
503 This feature has been disabled in Splunk configuration files.

Returned Values

Attribute Description
assureUTF8 Boolean value indicating wheter all data retreived from the index is proper UTF8.

If enabled (set to True), degrades indexing performance

Can only be set globally.

blockSignSize Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

blockSignatureDatabase The index that stores block signatures of events.

This is a global setting, not a per index setting.

bucketRebuildMemoryHint Suggestion for the Splunk bucket rebuild process for the size of the time-series (tsidx) file to make.
coldPath Filepath to the cold databases for the index.
coldPath_expanded Absoute filepath to the cold databases.
coldToFrozenDir Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

  • New style buckets (4.2 and on): removes all files but the rawdata
To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory
  • Old style buckets (Pre-4.2): gzip all the .data and .tsidx files
To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence.

coldToFrozenScript Path to the archiving script.

See the POST parameter description for details.

compressRawdata This value is ignored. splunkd process always compresses raw data.
currentDBSizeMB Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.
defaultDatabase If no index destination information is available in the input data, the index shown here is the destination of such data.
eai:acl See Access control lists for Splunk objects
enableOnlineBucketRepair Indicates whether to run asynchronous "online fsck" bucket repair, which runs in a process concurrently with Splunk.
enableRealtimeSearch Indicates if this is a real-time search.

This is a global setting, not a per index setting.

frozenTimePeriodInSecs Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath An absolute path that contains the hot and warm buckets for the index.
homePath_expanded An absolute filepath to the hot and warm buckets for the index.
indexThreads Number of threads used for indexing.

This is a global setting, not a per index setting.

isInternal Indicates if this is an internal index (for example, _internal, _audit).
lastInitTime Last time the index processor was successfully initialized.

This is a global setting, not a per index setting.

maxBloomBackfillBucketAge If a bucket (warm or cold) is older than this, Splunk does not create (or re-create) its bloom filter.
maxConcurrentOptimizes The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk to autotune this parameter (recommended). Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" is typically one that gets over 10GB of data per day.
  • "auto" sets the size to 750MB.
  • "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize will be auto tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

If a hot bucket exceeds maxHotIdleSecs, Splunk rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.

maxHotSpanSecs Upper bound of target maximum timespan of hot/warm buckets in seconds. Defaults to 7776000 seconds (90 days).

Note: If set too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB The amount of memory, in MB, allocated for indexing.

This is a global setting, not a per index setting.

maxMetaEntries Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you don't use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxTime UNIX timestamp of the newest event time in the index.
maxTimeUnreplicatedNoAcks Upper limit, in seconds, on how long an event can sit in raw slice. Applies only if replication is enabled for this index. Otherwise ignored.

If there are any acknowledged events sharing this raw slice, this paramater does not apply. In this case, maxTimeUnreplicatedWithAcks applies.

Highest legal value is 2147483647. To disable this parameter, set to 0.

Note: this is an advanced parameter. Understand the consequences before changing.

maxTimeUnreplicatedWithAcks Upper limit, in seconds, on how long events can sit unacknowledged in a raw slice. Applies only if you have enabled acks on forwarders and have replication enabled (with clustering).

Note: This is an advanced parameter. Make sure you understand the settings on all forwarders before changing this. This number should not exceed ack timeout configured on any forwarder, and should actually be set to at most half of the minimum value of that timeout. You can find this setting in outputs.conf readTimeout setting under the tcpout stanza.

To disable, set to 0, but this is NOT recommended. Highest legal value is 2147483647.

maxTotalDataSizeMB The maximum size of an index, in MB.
maxWarmDBCount The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.
memPoolMB Determines how much memory is given to the indexer memory pool.

This is a global setting, not a per-index setting.

minRawFileSyncSecs Can be either an integer (or "disable"). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

The integer sets how frequently splunkd forces a filesystem sync while compressing journal slices.

During this interval, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

minTime UNIX timestamp of the oldest event time in the index.
partialServiceMetaPeriod Related to serviceMetaPeriod. By default it is turned off (zero).

If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

quarantineFutureSecs Events with timestamp of quarantineFutureSecs newer than "now" are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

Warning: This is an advanced parameter. Only change it if instructed to do so by Splunk Support.

repFactor Value is either a non-negative number or "auto."

This attribute only applies to Splunk clustering slaves.

auto: Use the value as configured with the master.

0: Specify zero to turn off replication for this index.

For information on configuring clusters, see Configure clusters in the Splunk Managing Indexing and Clusters manual.

rotatePeriodInSecs Rotation period, in seconds, that specifies how frequently to check:
  • If a new hot bucket needs to be created.
  • If there are any cold buckets that should be frozen.
  • If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.
serviceMetaPeriod Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

suppressBannerList List of indexes for which we suppress "index missing" warning banner messages.

This is a global setting, not a per index setting.

sync Specifies the number of events that trigger the indexer to sync events.

This is a global setting, not a per index setting.

syncMeta When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of Splunk Support.

thawedPath Filepath to the thawed (resurrected) databases for the index.
thawedPath_expanded Absolute filepath to the thawed (resurrected) databases.
throttleCheckPeriod Defines how frequently Splunk checks for index throttling condition, in seconds. Defaults to 15 (seconds).

Note: Do not change this parameter without the input of Splunk Support.

totalEventCount Total number of events in the index.

Example

The following example creates an index named Shadow. 


curl -k -u admin:pass https://localhost:8089/servicesNS/admin/search/data/indexes \
	-d name=Shadow



<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:s="http://dev.splunk.com/ns/rest"
      xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
  <title>indexes</title>
  <id>https://localhost:8089/servicesNS/admin/search/data/indexes</id>
  <updated>2011-05-13T13:09:27-07:00</updated>
  <generator version="98392"/>
  <author>
    <name>Splunk</name>
  </author>
  <link href="/servicesNS/admin/search/data/indexes/_new" rel="create"/>
  <link href="/servicesNS/admin/search/data/indexes/_reload" rel="_reload"/>
  <opensearch:totalResults>1</opensearch:totalResults>
  <opensearch:itemsPerPage>30</opensearch:itemsPerPage>
  <opensearch:startIndex>0</opensearch:startIndex>
  <s:messages/>
  <entry>
    <title>shadow</title>
    <id>https://localhost:8089/servicesNS/nobody/search/data/indexes/shadow</id>
    <updated>2011-05-13T13:09:27-07:00</updated>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="alternate"/>
    <author>
      <name>nobody</name>
    </author>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="list"/>
    <link href="/servicesNS/nobody/search/data/indexes/shadow/_reload" rel="_reload"/>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="edit"/>
    <content type="text/xml">
      <s:dict>
        <s:key name="assureUTF8">0</s:key>
        <s:key name="blockSignSize">0</s:key>
        <s:key name="blockSignatureDatabase">_blocksignature</s:key>
        <s:key name="coldPath">$SPLUNK_DB/shadow/colddb</s:key>
        <s:key name="coldPath_expanded">/Applications/splunk/var/lib/splunk/shadow/colddb</s:key>
        <s:key name="coldToFrozenDir"></s:key>
        <s:key name="coldToFrozenScript"></s:key>
        <s:key name="compressRawdata">1</s:key>
        <s:key name="currentDBSizeMB">1</s:key>
        <s:key name="defaultDatabase">main</s:key>
        <s:key name="eai:acl">. . .</s:key>
        <s:key name="enableRealtimeSearch">1</s:key>
        <s:key name="frozenTimePeriodInSecs">188697600</s:key>
        <s:key name="homePath">$SPLUNK_DB/shadow/db</s:key>
        <s:key name="homePath_expanded">/Applications/splunk/var/lib/splunk/shadow/db</s:key>
        <s:key name="indexThreads">auto</s:key>
        <s:key name="isInternal">0</s:key>
        <s:key name="lastInitTime">1305317367.331268</s:key>
        <s:key name="maxConcurrentOptimizes">3</s:key>
        <s:key name="maxDataSize">auto</s:key>
        <s:key name="maxHotBuckets">3</s:key>
        <s:key name="maxHotIdleSecs">0</s:key>
        <s:key name="maxHotSpanSecs">7776000</s:key>
        <s:key name="maxMemMB">5</s:key>
        <s:key name="maxMetaEntries">1000000</s:key>
        <s:key name="maxTime"></s:key>
        <s:key name="maxTotalDataSizeMB">500000</s:key>
        <s:key name="maxWarmDBCount">300</s:key>
        <s:key name="memPoolMB">auto</s:key>
        <s:key name="minRawFileSyncSecs">disable</s:key>
        <s:key name="minTime"></s:key>
        <s:key name="partialServiceMetaPeriod">0</s:key>
        <s:key name="quarantineFutureSecs">2592000</s:key>
        <s:key name="quarantinePastSecs">77760000</s:key>
        <s:key name="rawChunkSizeBytes">131072</s:key>
        <s:key name="rotatePeriodInSecs">60</s:key>
        <s:key name="serviceMetaPeriod">25</s:key>
        <s:key name="suppressBannerList"></s:key>
        <s:key name="sync">0</s:key>
        <s:key name="syncMeta">1</s:key>
        <s:key name="thawedPath">$SPLUNK_DB/shadow/thaweddb</s:key>
        <s:key name="thawedPath_expanded">/Applications/splunk/var/lib/splunk/shadow/thaweddb</s:key>
        <s:key name="throttleCheckPeriod">15</s:key>
        <s:key name="totalEventCount">0</s:key>
      </s:dict>
    </content>
  </entry>
</feed>

data/indexes/{name}

DELETE data/indexes/{name}

Removes the index specified by {name} (not just the data contained in it).

Caution: This operation deletes the index's data directories and removes the index's stanza from indexes.conf.

Before executing this operation, look through all inputs.conf files (on your indexer and on any forwarders sending data to the indexer) and make sure that none of the stanzas are directing data to the index you plan to delete.

In other words, if you want to delete an index called "nogood," make sure the attribute/value pair index=nogood does not appear in any input stanzas. Once the index has been deleted, Splunk discards any data sent to that index.

For information on deleting indexes and deleting data from indexes, refer to Remove data from Splunk in the Splunk Managing Indexers and Clusters manual.

Request

No parameters for this request.

Response Codes

Status Code Description
200 Deleted successfully.
400 Request error. See response body for details.
401 Authentication failure: must pass valid credentials with request.
403 Insufficient permissions to delete the specified index.
404 Named index does not exist.
409 Request error: this operation is invalid for this item. See response body for details.
500 Internal server error. See response body for details.

Returned Values

No values returned for this request.

Example

Deletes the index named shadow. 

curl -k -u admin:pass --request DELETE https://localhost:8089/services/data/indexes/shadow

<feed xmlns="http://www.w3.org/2005/Atom" xmlns:s="http://dev.splunk.com/ns/rest" xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
  <title>indexes</title>
  <id>https://localhost:8089/services/data/indexes</id>
  <updated>2012-08-02T11:10:16-07:00</updated>
  <generator build="131547" version="5.0"/>
  <author>
    <name>Splunk</name>
  </author>
  <link href="/services/data/indexes/_new" rel="create"/>
  <link href="/services/data/indexes/_reload" rel="_reload"/>
  <opensearch:totalResults>0</opensearch:totalResults>
  <opensearch:itemsPerPage>30</opensearch:itemsPerPage>
  <opensearch:startIndex>0</opensearch:startIndex>
  <s:messages/>
</feed>

GET data/indexes/{name}

Retrieves information about the named index.

Request

Name Type Required Default Description
summarize Bool If true, leaves out certain index details in order to provide a faster response.

Response Codes

Status Code Description
200 Listed successfully.
400 Request error. See response body for details.
401 Authentication failure: must pass valid credentials with request.
403 Insufficient permissions to view index.
404 Index does not exist.
409 Request error: this operation is invalid for this item. See response body for details.
500 Internal server error. See response body for details.

Returned Values

Attribute Description
assureUTF8 Indicates whether all data retreived from the index is proper UTF8. If enabled (set to True), degrades indexing performance.

This is a global setting, not a per index setting.

blockSignSize Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

blockSignatureDatabase The index that stores block signatures of events.

This is a global setting, not a per index setting.

bloomfilterTotalSizeKB Total size of all bloom filter files, in KB.
coldPath Filepath to the cold databases for the index.
coldPath_expanded Absoute filepath to the cold databases.
coldToFrozenDir Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

  • New style buckets (4.2 and on): removes all files but the rawdata
To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory
  • Old style buckets (Pre-4.2): gzip all the .data and .tsidx files
To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence.

coldToFrozenScript Path to the archiving script.

See the POST parameter description for details.

compressRawdata This value is ignored. splunkd process always compresses raw data.
currentDBSizeMB Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.
defaultDatabase If no index destination information is available in the input data, the index shown here is the destination of such data.
disabled Indicates if the index is disabled.
eai:attributes See Accessing Splunk resources
enableRealtimeSearch Indicates if this is a real-time search.

This is a global setting, not a per index setting.

frozenTimePeriodInSecs Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath An absolute path that contains the hot and warm buckets for the index.
homePath_expanded An absolute filepath to the hot and warm buckets for the index.
indexThreads Number of threads used for indexing.

This is a global setting, not a per index setting.

isInternal Indicates if this is an internal index (for example, _internal, _audit).
lastInitTime Last time the index processor was successfully initialized.

This is a global setting, not a per index setting.

maxConcurrentOptimizes The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk to autotune this parameter (recommended). Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" is typically one that gets over 10GB of data per day.
  • "auto" sets the size to 750MB.
  • "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize will be auto tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

If a hot bucket exceeds maxHotIdleSecs, Splunk rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.

maxHotSpanSecs Upper bound of target maximum timespan of hot/warm buckets in seconds. Defaults to 7776000 seconds (90 days).

Note: If set too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB The amount of memory, in MB, allocated for indexing.

This is a global setting, not a per index setting.

maxMetaEntries Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you don't use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxRunningProcessGroups Maximum number of processes that the indexer fires off at a time.

This is a global setting, not a per index setting.

maxTime UNIX timestamp of the newest event time in the index.
maxTotalDataSizeMB The maximum size of an index, in MB.
maxWarmDBCount The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.
memPoolMB Determines how much memory is given to the indexer memory pool.

This is a global setting, not a per-index setting.

minRawFileSyncSecs Can be either an integer (or "disable"). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

The integer sets how frequently splunkd forces a filesystem sync while compressing journal slices.

During this interval, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

minTime UNIX timestamp of the oldest event time in the index.
numBloomfilters The number of bloom filters created for this index.
numHotBuckets The number of hot buckets created for this index.
numWarmBuckets The number of warm buckets created for this index.
partialServiceMetaPeriod Related to serviceMetaPeriod. By default it is turned off (zero).

If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

quarantineFutureSecs Events with timestamp of quarantineFutureSecs newer than "now" that are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

Warning: This is an advanced parameter. Only change it if instructed to do so by Splunk Support.

rotatePeriodInSecs Rotation period, in seconds, that specifies how frequently to check:
  • If a new hot bucket needs to be created.
  • If there are any cold buckets that should be frozen.
  • If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.
serviceMetaPeriod Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

summarize If true, leaves out certain index details, which provides a faster response.
suppressBannerList List of indexes for which we suppress "index missing" warning banner messages.

This is a global setting, not a per index setting.

sync Specifies the number of events that trigger the indexer to sync events.

This is a global setting, not a per index setting.

syncMeta When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of Splunk Support.

thawedPath An absolute path that contains the thawed (resurrected) databases for the index.
thawedPath_expanded Absolute filepath to the thawed (resurrected) databases.
throttleCheckPeriod Defines how frequently Splunk checks for index throttling condition, in seconds. Defaults to 15 (seconds).

Note: Do not change this parameter without the input of Splunk Support.

totalEventCount Total number of events in the index.

Example

Lists information about the Shadow index. 


curl -k -u admin:pass https://localhost:8089/servicesNS/nobody/search/data/indexes/shadow



<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"
      xmlns:s="http://dev.splunk.com/ns/rest">
  <title>indexes</title>
  <id>https://localhost:8089/servicesNS/nobody/search/data/indexes</id>
  <updated>2011-08-01T12:25:34-07:00</updated>
  <generator version="105103"/>
  <author>
    <name>Splunk</name>
  </author>
  <link href="/servicesNS/nobody/search/data/indexes/_new" rel="create"/>
  <link href="/servicesNS/nobody/search/data/indexes/_reload" rel="_reload"/>
  <!-- opensearch nodes elided for brevity. -->
  <s:messages/>
  <entry>
    <title>shadow</title>
    <id>https://localhost:8089/servicesNS/nobody/search/data/indexes/shadow</id>
    <updated>2011-08-01T11:47:55-07:00</updated>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="alternate"/>
    <author>
      <name>nobody</name>
    </author>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="list"/>
    <link href="/servicesNS/nobody/search/data/indexes/shadow/_reload" rel="_reload"/>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="edit"/>
    <link href="/servicesNS/nobody/search/data/indexes/shadow/disable" rel="disable"/>
    <content type="text/xml">
      <s:dict>
        <s:key name="assureUTF8">0</s:key>
        <s:key name="blockSignSize">0</s:key>
        <s:key name="blockSignatureDatabase">_blocksignature</s:key>
        <s:key name="bloomfilterTotalSizeKB">0</s:key>
        <s:key name="coldPath">$SPLUNK_DB/shadow/colddb</s:key>
        <s:key name="coldPath_expanded">/home/amrit/bin/splunk-current/var/lib/splunk/shadow/colddb</s:key>
        <s:key name="coldToFrozenDir"/>
        <s:key name="coldToFrozenScript"/>
        <s:key name="compressRawdata">1</s:key>
        <s:key name="currentDBSizeMB">1</s:key>
        <s:key name="defaultDatabase">main</s:key>
        <s:key name="disabled">0</s:key>
        <!-- eai:acl nodes elided for brevity. -->
        <s:key name="eai:attributes">
          <s:dict>
            <s:key name="optionalFields">
              <s:list>
                <s:item>assureUTF8</s:item>
                <s:item>blockSignSize</s:item>
                <s:item>coldToFrozenDir</s:item>
                <s:item>coldToFrozenScript</s:item>
                <s:item>compressRawdata</s:item>
                <s:item>frozenTimePeriodInSecs</s:item>
                <s:item>maxConcurrentOptimizes</s:item>
                <s:item>maxDataSize</s:item>
                <s:item>maxHotBuckets</s:item>
                <s:item>maxHotIdleSecs</s:item>
                <s:item>maxHotSpanSecs</s:item>
                <s:item>maxMemMB</s:item>
                <s:item>maxMetaEntries</s:item>
                <s:item>maxRunningProcessGroups</s:item>
                <s:item>maxTotalDataSizeMB</s:item>
                <s:item>maxWarmDBCount</s:item>
                <s:item>minRawFileSyncSecs</s:item>
                <s:item>partialServiceMetaPeriod</s:item>
                <s:item>quarantineFutureSecs</s:item>
                <s:item>quarantinePastSecs</s:item>
                <s:item>rawChunkSizeBytes</s:item>
                <s:item>rotatePeriodInSecs</s:item>
                <s:item>serviceMetaPeriod</s:item>
                <s:item>suppressBannerList</s:item>
                <s:item>syncMeta</s:item>
                <s:item>throttleCheckPeriod</s:item>
              </s:list>
            </s:key>
            <s:key name="requiredFields">
              <s:list/>
            </s:key>
            <s:key name="wildcardFields">
              <s:list/>
            </s:key>
          </s:dict>
        </s:key>
        <s:key name="enableRealtimeSearch">1</s:key>
        <s:key name="frozenTimePeriodInSecs">188697600</s:key>
        <s:key name="homePath">$SPLUNK_DB/shadow/db</s:key>
        <s:key name="homePath_expanded">/home/amrit/bin/splunk-current/var/lib/splunk/shadow/db</s:key>
        <s:key name="indexThreads">auto</s:key>
        <s:key name="isInternal">0</s:key>
        <s:key name="lastInitTime">1312226552.102920</s:key>
        <s:key name="maxConcurrentOptimizes">3</s:key>
        <s:key name="maxDataSize">auto</s:key>
        <s:key name="maxHotBuckets">3</s:key>
        <s:key name="maxHotIdleSecs">0</s:key>
        <s:key name="maxHotSpanSecs">7776000</s:key>
        <s:key name="maxMemMB">5</s:key>
        <s:key name="maxMetaEntries">1000000</s:key>
        <s:key name="maxRunningProcessGroups">20</s:key>
        <s:key name="maxTime"/>
        <s:key name="maxTotalDataSizeMB">500000</s:key>
        <s:key name="maxWarmDBCount">300</s:key>
        <s:key name="memPoolMB">auto</s:key>
        <s:key name="minRawFileSyncSecs">disable</s:key>
        <s:key name="minTime"/>
        <s:key name="numBloomfilters">0</s:key>
        <s:key name="numHotBuckets">0</s:key>
        <s:key name="numWarmBuckets">0</s:key>
        <s:key name="partialServiceMetaPeriod">0</s:key>
        <s:key name="quarantineFutureSecs">2592000</s:key>
        <s:key name="quarantinePastSecs">77760000</s:key>
        <s:key name="rawChunkSizeBytes">131072</s:key>
        <s:key name="rotatePeriodInSecs">60</s:key>
        <s:key name="serviceMetaPeriod">25</s:key>
        <s:key name="suppressBannerList"/>
        <s:key name="sync">0</s:key>
        <s:key name="syncMeta">1</s:key>
        <s:key name="thawedPath">$SPLUNK_DB/shadow/thaweddb</s:key>
        <s:key name="thawedPath_expanded">/home/amrit/bin/splunk-current/var/lib/splunk/shadow/thaweddb</s:key>
        <s:key name="throttleCheckPeriod">15</s:key>
        <s:key name="totalEventCount">0</s:key>
      </s:dict>
    </content>
  </entry>
</feed>

POST data/indexes/{name}

Updates the data index specified by {name} with information specified with index attributes.

Request

Name Type Required Default Description
blockSignSize Number 0 Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

bucketRebuildMemoryHint String auto Suggestion for the Splunk bucket rebuild process for the size of the time-series (tsidx) file to make.

Caution: This is an advanced parameter. Inappropriate use of this parameter causes splunkd to not start if rebuild is required. Do not set this parameter unless instructed by Splunk Support.

Default value, auto, varies by the amount of physical RAM on the host

  • less than 2GB RAM = 67108864 (64MB) tsidx
  • 2GB to 8GB RAM = 134217728 (128MB) tsidx
  • more than 8GB RAM = 268435456 (256MB) tsidx

Values other than "auto" must be 16MB-1GB. Highest legal value (of the numerical part) is 4294967295

You can specify the value using a size suffix: "16777216" or "16MB" are equivalent.

coldToFrozenDir String Destination path for the frozen archive. Use as an alternative to a coldToFrozenScript. Splunk automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

  • New style buckets (4.2 and on): removes all files but the rawdata
To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory
  • Old style buckets (Pre-4.2): gzip all the .data and .tsidx files
To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence

coldToFrozenScript String Path to the archiving script.

If your script requires a program to run it (for example, python), specify the program followed by the path. The script must be in $SPLUNK_HOME/bin or one of its subdirectories.

Splunk ships with an example archiving script in $SPLUNK_HOME/bin called coldToFrozenExample.py. Splunk DOES NOT recommend using this example script directly. It uses a default path, and if modified in place any changes will be overwritten on upgrade.

Splunk recommends copying the example script to a new file in bin and modifying it for your system. Most importantly, change the default archive path to an existing directory that fits your needs.

If your new script in bin/ is named myColdToFrozen.py, set this key to the following:

coldToFrozenScript = "$SPLUNK_HOME/bin/python" "$SPLUNK_HOME/bin/myColdToFrozen.py"

By default, the example script has two possible behaviors when archiving:

  • For buckets created from version 4.2 and on, it removes all files except for rawdata. To thaw: cd to the frozen bucket and type splunk rebuild ., then copy the bucket to thawed for that index. We recommend using the coldToFrozenDir parameter unless you need to perform a more advanced operation upon freezing buckets.
  • For older-style buckets, we simply gzip all the .tsidx files. To thaw: cd to the frozen bucket and unzip the tsidx files, then copy the bucket to thawed for that index
compressRawdata Boolean true This parameter is ignored. The splunkd process always compresses raw data.
enableOnlineBucketRepair Boolean true Enables asynchronous "online fsck" bucket repair, which runs concurrently with Splunk.

When enabled, you do not have to wait until buckets are repaired to start Splunk. However, you might observe a slight performance degratation.

Note: This endpoint is new in Splunk 4.3.

frozenTimePeriodInSecs Number 188697600 Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

maxBloomBackfillBucketAge Number 30d Valid values are: Integer[m|s|h|d]

If a warm or cold bucket is older than the specified age, do not create or rebuild its bloomfilter. Specify 0 to never rebuild bloomfilters.

For example, if a bucket is older than specified with maxBloomBackfillBucketAge, and the rebuilding of its bloomfilter started but did not finish, do not rebuild it.

maxConcurrentOptimizes Number 3 The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize Number auto The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk to autotune this parameter (recommended).Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" would typically be considered one that gets over 10GB of data per day.
  • "auto" sets the size to 750MB.
  • "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize will be auto tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets Number 3 Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs Number 0 Maximum life, in seconds, of a hot bucket. Defaults to 0.

If a hot bucket exceeds maxHotIdleSecs, Splunk rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

maxHotSpanSecs Number 7776000 Upper bound of target maximum timespan of hot/warm buckets in seconds. Defaults to 7776000 seconds (90 days).

Note:I f you set this too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB Number 5 The amount of memory, expressed in MB, to allocate for buffering a single tsidx file into memory before flushing to disk. Defaults to 5. The default is recommended for all environments.

IMPORTANT: Calculate this number carefully. Setting this number incorrectly may have adverse effects on your systems memory and/or splunkd stability/performance.

maxMetaEntries Number 1000000 Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you don't use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxTimeUnreplicatedNoAcks Number 300 Upper limit, in seconds, on how long an event can sit in raw slice. Applies only if replication is enabled for this index. Otherwise ignored.

If there are any acknowledged events sharing this raw slice, this paramater does not apply. In this case, maxTimeUnreplicatedWithAcks applies.

Highest legal value is 2147483647. To disable this parameter, set to 0.

Note: this is an advanced parameter. Understand the consequences before changing.

maxTimeUnreplicatedWithAcks Number 60 Upper limit, in seconds, on how long events can sit unacknowledged in a raw slice. Applies only if you have enabled acks on forwarders and have replication enabled (with clustering).

Note: This is an advanced parameter. Make sure you understand the settings on all forwarders before changing this. This number should not exceed ack timeout configured on any forwarder, and should actually be set to at most half of the minimum value of that timeout. You can find this setting in outputs.conf readTimeout setting under the tcpout stanza.

To disable, set to 0, but this is NOT recommended. Highest legal value is 2147483647.

maxTotalDataSizeMB Number 500000 The maximum size of an index (in MB). If an index grows larger than the maximum size, the oldest data is frozen.
maxWarmDBCount Number 300 The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times will be moved to cold.
minRawFileSyncSecs Number disable Specify an integer (or "disable") for this parameter.

This parameter sets how frequently splunkd forces a filesystem sync while compressing journal slices.

During this interval, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

Note: Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

partialServiceMetaPeriod Number 0 Related to serviceMetaPeriod. If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

By default it is turned off (zero).

quarantineFutureSecs Number 2592000 Events with timestamp of quarantineFutureSecs newer than "now" are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs Number 77760000 Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes Number 131072 Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

WARNING: This is an advanced parameter. Only change it if you are instructed to do so by Splunk Support.

repFactor String 0 Value is either a non-negative number or "auto." This parameter only applies to Splunk clustering slaves.

auto: Use the value as configured with the master.

0: Specify zero to turn off replication for this index.

For information on configuring clusters, see Configure clusters in the Splunk Managing Indexing and Clusters manual.

rotatePeriodInSecs Number 60 How frequently (in seconds) to check if a new hot bucket needs to be created. Also, how frequently to check if there are any warm/cold buckets that should be rolled/frozen.
serviceMetaPeriod Number 25 Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

syncMeta Boolean true When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of a Splunk Support.

throttleCheckPeriod Number 15 Defines how frequently Splunk checks for index throttling condition, in seconds. Defaults to 15 (seconds).

Note: Do not change this parameter without the input of Splunk Support.

Response Codes

Status Code Description
200 Properties for the index were updated successfully.
400 Some arguments were invalid
401 Authentication failure: must pass valid credentials with request.
402 The Splunk license in use has disabled this feature.
403 Insufficient permissions to edit index.
404 The specified index was not found.
409 Request error: this operation is invalid for this item. See response body for details.
500 Unspecified error
503 This feature has been disabled in Splunk configuration files.

Returned Values

Attribute Description
assureUTF8 Boolean value indicating wheter all data retreived from the index is proper UTF8.

If enabled (set to True), degrades indexing performance

Can only be set globally.

blockSignSize Controls how many events make up a block for block signatures.

If this is set to 0, block signing is disabled for this index.

A recommended value is 100.

blockSignatureDatabase The index that stores block signatures of events.

This is a global setting, not a per index setting.

bucketRebuildMemoryHint Suggestion for the Splunk bucket rebuild process for the size of the time-series (tsidx) file to make.
coldPath Filepath to the cold databases for the index.
coldPath_expanded Absoute filepath to the cold databases.
coldToFrozenDir Destination path for the frozen archive. Used as an alternative to a coldToFrozenScript. Splunk automatically puts frozen buckets in this directory.

Bucket freezing policy is as follows:

  • New style buckets (4.2 and on): removes all files but the rawdata
To thaw, run splunk rebuild <bucket dir> on the bucket, then move to the thawed directory
  • Old style buckets (Pre-4.2): gzip all the .data and .tsidx files
To thaw, gunzip the zipped files and move the bucket into the thawed directory

If both coldToFrozenDir and coldToFrozenScript are specified, coldToFrozenDir takes precedence.

coldToFrozenScript Path to the archiving script.

See the POST parameter description for details.

compressRawdata This value is ignored. splunkd process always compresses raw data.
currentDBSizeMB Total size, in MB, of data stored in the index. The total incudes data in the home, cold and thawed paths.
defaultDatabase If no index destination information is available in the input data, the index shown here is the destination of such data.
eai:acl See Access control lists for Splunk objects
enableOnlineBucketRepair Indicates whether to run asynchronous "online fsck" bucket repair, which runs in a process concurrently with Splunk.
enableRealtimeSearch Indicates if this is a real-time search.

This is a global setting, not a per index setting.

frozenTimePeriodInSecs Number of seconds after which indexed data rolls to frozen. Defaults to 188697600 (6 years).

Freezing data means it is removed from the index. If you need to archive your data, refer to coldToFrozenDir and coldToFrozenScript parameter documentation.

homePath An absolute path that contains the hot and warm buckets for the index.
homePath_expanded An absolute filepath to the hot and warm buckets for the index.
indexThreads Number of threads used for indexing.

This is a global setting, not a per index setting.

isInternal Indicates if this is an internal index (for example, _internal, _audit).
lastInitTime Last time the index processor was successfully initialized.

This is a global setting, not a per index setting.

maxBloomBackfillBucketAge If a bucket (warm or cold) is older than this, Splunk does not create (or re-create) its bloom filter.
maxConcurrentOptimizes The number of concurrent optimize processes that can run against a hot bucket.

This number should be increased if instructed by Splunk Support. Typically the default value should suffice.

maxDataSize The maximum size in MB for a hot DB to reach before a roll to warm is triggered. Specifying "auto" or "auto_high_volume" causes Splunk to autotune this parameter (recommended). Use "auto_high_volume" for high volume indexes (such as the main index); otherwise, use "auto". A "high volume index" is typically one that gets over 10GB of data per day.
  • "auto" sets the size to 750MB.
  • "auto_high_volume" sets the size to 10GB on 64-bit, and 1GB on 32-bit systems.

Although the maximum value you can set this is 1048576 MB, which corresponds to 1 TB, a reasonable number ranges anywhere from 100 - 50000. Any number outside this range should be approved by Splunk Support before proceeding.

If you specify an invalid number or string, maxDataSize will be auto tuned.

Note: The precise size of your warm buckets may vary from maxDataSize, due to post-processing and timing issues with the rolling policy.

maxHotBuckets Maximum hot buckets that can exist per index. Defaults to 3.

When maxHotBuckets is exceeded, Splunk rolls the least recently used (LRU) hot bucket to warm. Both normal hot buckets and quarantined hot buckets count towards this total. This setting operates independently of maxHotIdleSecs, which can also cause hot buckets to roll.

maxHotIdleSecs Maximum life, in seconds, of a hot bucket. Defaults to 0. A value of 0 turns off the idle check (equivalent to INFINITE idle time).

If a hot bucket exceeds maxHotIdleSecs, Splunk rolls it to warm. This setting operates independently of maxHotBuckets, which can also cause hot buckets to roll.

maxHotSpanSecs Upper bound of target maximum timespan of hot/warm buckets in seconds. Defaults to 7776000 seconds (90 days).

Note: If set too small, you can get an explosion of hot/warm buckets in the filesystem. The system sets a lower bound implicitly for this parameter at 3600, but this is an advanced parameter that should be set with care and understanding of the characteristics of your data.

maxMemMB The amount of memory, in MB, allocated for indexing.

This is a global setting, not a per index setting.

maxMetaEntries Sets the maximum number of unique lines in .data files in a bucket, which may help to reduce memory consumption. If set to 0, this setting is ignored (it is treated as infinite).

If exceeded, a hot bucket is rolled to prevent further increase. If your buckets are rolling due to Strings.data hitting this limit, the culprit may be the punct field in your data. If you don't use punct, it may be best to simply disable this (see props.conf.spec in $SPLUNK_HOME/etc/system/README).

There is a small time delta between when maximum is exceeded and bucket is rolled. This means a bucket may end up with epsilon more lines than specified, but this is not a major concern unless excess is significant.

maxTime UNIX timestamp of the newest event time in the index.
maxTimeUnreplicatedNoAcks Upper limit, in seconds, on how long an event can sit in raw slice. Applies only if replication is enabled for this index. Otherwise ignored.

If there are any acknowledged events sharing this raw slice, this paramater does not apply. In this case, maxTimeUnreplicatedWithAcks applies.

Highest legal value is 2147483647. To disable this parameter, set to 0.

Note: this is an advanced parameter. Understand the consequences before changing.

maxTimeUnreplicatedWithAcks Upper limit, in seconds, on how long events can sit unacknowledged in a raw slice. Applies only if you have enabled acks on forwarders and have replication enabled (with clustering).

Note: This is an advanced parameter. Make sure you understand the settings on all forwarders before changing this. This number should not exceed ack timeout configured on any forwarder, and should actually be set to at most half of the minimum value of that timeout. You can find this setting in outputs.conf readTimeout setting under the tcpout stanza.

To disable, set to 0, but this is NOT recommended. Highest legal value is 2147483647.

maxTotalDataSizeMB The maximum size of an index, in MB.
maxWarmDBCount The maximum number of warm buckets. If this number is exceeded, the warm bucket/s with the lowest value for their latest times are moved to cold.
memPoolMB Determines how much memory is given to the indexer memory pool.

This is a global setting, not a per-index setting.

minRawFileSyncSecs Can be either an integer (or "disable"). Some filesystems are very inefficient at performing sync operations, so only enable this if you are sure it is needed

The integer sets how frequently splunkd forces a filesystem sync while compressing journal slices.

During this interval, uncompressed slices are left on disk even after they are compressed. Then splunkd forces a filesystem sync of the compressed journal and removes the accumulated uncompressed files.

If 0 is specified, splunkd forces a filesystem sync after every slice completes compressing. Specifying "disable" disables syncing entirely: uncompressed slices are removed as soon as compression is complete.

minTime UNIX timestamp of the oldest event time in the index.
partialServiceMetaPeriod Related to serviceMetaPeriod. By default it is turned off (zero).

If set, it enables metadata sync every <integer> seconds, but only for records where the sync can be done efficiently in-place, without requiring a full re-write of the metadata file. Records that require full re-write are be sync'ed at serviceMetaPeriod.

partialServiceMetaPeriod specifies, in seconds, how frequently it should sync. Zero means that this feature is turned off and serviceMetaPeriod is the only time when metadata sync happens.

If the value of partialServiceMetaPeriod is greater than serviceMetaPeriod, this setting has no effect.

quarantineFutureSecs Events with timestamp of quarantineFutureSecs newer than "now" are dropped into quarantine bucket. Defaults to 2592000 (30 days).

This is a mechanism to prevent main hot buckets from being polluted with fringe events.

quarantinePastSecs Events with timestamp of quarantinePastSecs older than "now" are dropped into quarantine bucket. Defaults to 77760000 (900 days).

This is a mechanism to prevent the main hot buckets from being polluted with fringe events.

rawChunkSizeBytes Target uncompressed size in bytes for individual raw slice in the rawdata journal of the index. Defaults to 131072 (128KB). 0 is not a valid value. If 0 is specified, rawChunkSizeBytes is set to the default value.

Note: rawChunkSizeBytes only specifies a target chunk size. The actual chunk size may be slightly larger by an amount proportional to an individual event size.

Warning: This is an advanced parameter. Only change it if instructed to do so by Splunk Support.

repFactor Value is either a non-negative number or "auto."

This attribute only applies to Splunk clustering slaves.

auto: Use the value as configured with the master.

0: Specify zero to turn off replication for this index.

For information on configuring clusters, see Configure clusters in the Splunk Managing Indexing and Clusters manual.

rotatePeriodInSecs Rotation period, in seconds, that specifies how frequently to check:
  • If a new hot bucket needs to be created.
  • If there are any cold buckets that should be frozen.
  • If there are any buckets that need to be moved out hot and cold DBs, due to size constraints.
serviceMetaPeriod Defines how frequently metadata is synced to disk, in seconds. Defaults to 25 (seconds).

You may want to set this to a higher value if the sum of your metadata file sizes is larger than many tens of megabytes, to avoid the hit on I/O in the indexing fast path.

suppressBannerList List of indexes for which we suppress "index missing" warning banner messages.

This is a global setting, not a per index setting.

sync Specifies the number of events that trigger the indexer to sync events.

This is a global setting, not a per index setting.

syncMeta When true, a sync operation is called before file descriptor is closed on metadata file updates. This functionality improves integrity of metadata files, especially in regards to operating system crashes/machine failures.

Note: Do not change this parameter without the input of Splunk Support.

thawedPath Filepath to the thawed (resurrected) databases for the index.
thawedPath_expanded Absolute filepath to the thawed (resurrected) databases.
throttleCheckPeriod Defines how frequently Splunk checks for index throttling condition, in seconds. Defaults to 15 (seconds).

Note: Do not change this parameter without the input of Splunk Support.

totalEventCount Total number of events in the index.

Example

The following example updates the maximum size of the index named "Shadow", setting the size at 400000 MB.

This index was created in the example for the POST operation for this endpoint. 


curl -k -u admin:pass \
	-d maxTotalDataSizeMB=400000 https://localhost:8089/servicesNS/nobody/search/data/indexes/shadow



<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:s="http://dev.splunk.com/ns/rest"
      xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
  <title>indexes</title>
  <id>https://localhost:8089/servicesNS/nobody/search/data/indexes</id>
  <updated>2011-05-16T12:20:06-07:00</updated>
  <generator version="98392"/>
  <author>
    <name>Splunk</name>
  </author>
  <link href="/servicesNS/nobody/search/data/indexes/_new" rel="create"/>
  <link href="/servicesNS/nobody/search/data/indexes/_reload" rel="_reload"/>
  <opensearch:totalResults>1</opensearch:totalResults>
  <opensearch:itemsPerPage>30</opensearch:itemsPerPage>
  <opensearch:startIndex>0</opensearch:startIndex>
  <s:messages/>
  <entry>
    <title>shadow</title>
    <id>https://localhost:8089/servicesNS/nobody/search/data/indexes/shadow</id>
    <updated>2011-05-16T12:18:56-07:00</updated>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="alternate"/>
    <author>
      <name>nobody</name>
    </author>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="list"/>
    <link href="/servicesNS/nobody/search/data/indexes/shadow/_reload" rel="_reload"/>
    <link href="/servicesNS/nobody/search/data/indexes/shadow" rel="edit"/>
    <content type="text/xml">
      <s:dict>
        <s:key name="assureUTF8">0</s:key>
        <s:key name="blockSignSize">0</s:key>
        <s:key name="blockSignatureDatabase">_blocksignature</s:key>
        <s:key name="coldPath">$SPLUNK_DB/shadow/colddb</s:key>
        <s:key name="coldPath_expanded">/Applications/splunk4.3/var/lib/splunk/shadow/colddb</s:key>
        <s:key name="coldToFrozenDir"></s:key>
        <s:key name="coldToFrozenScript"></s:key>
        <s:key name="compressRawdata">1</s:key>
        <s:key name="currentDBSizeMB">1</s:key>
        <s:key name="defaultDatabase">main</s:key>
        <s:key name="eai:acl">. . .</s:key>
        <s:key name="enableRealtimeSearch">1</s:key>
        <s:key name="frozenTimePeriodInSecs">188697600</s:key>
        <s:key name="homePath">$SPLUNK_DB/shadow/db</s:key>
        <s:key name="homePath_expanded">/Applications/splunk4.3/var/lib/splunk/shadow/db</s:key>
        <s:key name="indexThreads">auto</s:key>
        <s:key name="isInternal">0</s:key>
        <s:key name="lastInitTime">1305573611.118477</s:key>
        <s:key name="maxConcurrentOptimizes">3</s:key>
        <s:key name="maxDataSize">auto</s:key>
        <s:key name="maxHotBuckets">3</s:key>
        <s:key name="maxHotIdleSecs">0</s:key>
        <s:key name="maxHotSpanSecs">7776000</s:key>
        <s:key name="maxMemMB">5</s:key>
        <s:key name="maxMetaEntries">1000000</s:key>
        <s:key name="maxTime"></s:key>
        <s:key name="maxTotalDataSizeMB">400000</s:key>
        <s:key name="maxWarmDBCount">300</s:key>
        <s:key name="memPoolMB">auto</s:key>
        <s:key name="minRawFileSyncSecs">disable</s:key>
        <s:key name="minTime"></s:key>
        <s:key name="partialServiceMetaPeriod">0</s:key>
        <s:key name="quarantineFutureSecs">2592000</s:key>
        <s:key name="quarantinePastSecs">77760000</s:key>
        <s:key name="rawChunkSizeBytes">131072</s:key>
        <s:key name="rotatePeriodInSecs">60</s:key>
        <s:key name="serviceMetaPeriod">25</s:key>
        <s:key name="suppressBannerList"></s:key>
        <s:key name="sync">0</s:key>
        <s:key name="syncMeta">1</s:key>
        <s:key name="thawedPath">$SPLUNK_DB/shadow/thaweddb</s:key>
        <s:key name="thawedPath_expanded">/Applications/splunk4.3/var/lib/splunk/shadow/thaweddb</s:key>
        <s:key name="throttleCheckPeriod">15</s:key>
        <s:key name="totalEventCount">0</s:key>
      </s:dict>
    </content>
  </entry>
</feed>
Retrieved from "http://docs.splunk.com/Documentation/Splunk/5.0.2/RESTAPI/RESTindex"

This documentation applies to the following versions of Splunk: 5.0 , 5.0.1 , 5.0.2 View the Article History for its revisions.


You must be logged into splunk.com in order to post comments. Log in now.

Was this documentation topic helpful?

If you'd like to hear back from us, please provide your email address:

We'd love to hear what you think about this topic or the documentation as a whole. Feedback you enter here will be delivered to the documentation team.

Feedback submitted, thanks!