
KV store endpoint descriptions
This introduction describes syntax common to all App KV Store REST endpoints. For a description of how to use the App KV Store REST API, see App Key Value Store.
Note: Username and password authentication is required for most endpoints and REST operations. Additional capability or role-based authorization may also be required, particularly for POST or DELETE operations.
App KV store REST API features
App KV store provides:
- Create, read, list and delete EAI operations on a collection.
- Indices create, read, list and delete indices, for a given collection. All collection methods are EAI-compatible. These are not regular EAI operations because the indices are stored as attributes on the EAI entity. If a user does not have a permission for a given collection, these operations are prohibited.
- The ability to insert, read, update and delete a single key and key value, for a given collection, provided you have permission to modify the collection.
- Duplicate keys are not allowed.
- All updates are wholesale updates.
- Partial value update is prohibited.
- The ability to query key-value pairs, for a given collection. For example, to find all the documents that have a host field with a value, use the query {"host": "bar.com"}. Only the following operations are permitted:
$gt
,$gte
,$lt
,$lte
,$ne
,$and
,$or
, and$not
. The API also supports mechanisms for projection, sorting and pagination. - The ability to do batch reads/inserts/updates, for a given collection. This supports high-volume operations and limits network traffic. These operations have the same semantics as their individual counterparts.
Limits
- 16MB per record
- 1KB per indexed field
Collections
A Collection is a container of "documents" or "values". Most documents have a similar structure, although structure is not enforced. For example, a Collection might have Notable Events
or entities
.
Collections are declared in a collections.conf
configuration file, where a stanza is a single collection.
Collections are Splunk platform entities, which means they are permissioned using the Splunk platform RBAC/ACL system. One reason of exposing Collections as entities is to ensure they are properly managed. But, the implementation permits only app-level collections. This means that collections can be defined only in the etc/apps/<app>/<default|local>
directory, not in etc/users/...
. Attempts to create a user-specific Collection using the API fail and Collections created manually in the .conf
files are ignored.
Data
A collection stores key-value pairs consisting of a user-specified or autogenerated key and a JSON document-formatted value. A JSON document/object of values is enforced for simplicity instead of using integer or string raw values. For example, {"_key": "10.0.0.1", host: "bar.com"}
defines the key ("10.0.0.1"
) and the full JSON document value.
A key is fetched, updated, and deleted. Partial updates, including array append, are prohibited. You must set the whole document. Keys are unique within a collection and support only the basic JSON data types: boolean, string, number, object, array, and null
.
A value is specified in the _key
attribute of the document and, if one is not specified, a random value with a desirable sort order is automatically generated.
A collection defined at:
/servicesNS/nobody/search/storage/mycollection
can have operations performed on it using the following URLs:
/servicesNS/nobody/search/storage/mycollection
/servicesNS/itay/search/storage/mycollection
/servicesNS/mark/search/storage/mycollection
Namespaces are separate, and data are not interleaved. Wildcard operations on the <user>
or <app>
parts of the namespace are prohibited.
User-specific collections are not supported so all users, including nobody
, share a single namespace and _key
is unique across all namespaces.
Types
Defining a set of types for some fields in a collection is provided. If a field has no type defined, it uses the type submitted over the API. If a type conversion fails, the insert/update operation is aborted.
For example, if the collections.conf
file has the following collection
[mycollection] field.enabled = bool field.data.range = cidr
the data inserted or updated in the collection have the fields enabled and range-converted to the applicable type:
* array * number * boolean * time * string * cidr
Items need to specified using the field
prefix only if type needs to be enforced for the field. The array
type and cidr
types are not enforceable, so they do not need to be specified.
Arrays
Arrays do not change a nested key name. For example, the key "a.b.c" can refer to {"a": {"b": {"c": 1}}}
or {"a": {"b": [{"c": 1}]}}
.
Numbers
All strings/numbers/booleans are converted to doubles. If a number fails to convert, it inserts the string instead.
Booleans
All numbers/strings/booleans are converted to booleans. If a boolean fails to convert, it inserts the string instead. Epoch time can be a string or as a number. If it fails to convert, the string version is inserted.
Strings
Everything is converted to a string.
CIDR
All provided strings are converted to canonical CIDR strings. For example:
127.0.0.1 -> 127.000.000.001/32 127.0.0.0/24 -> 127.000.000.000/24 127.0.0/24 -> 127.000.000.000/24
with similar canonicalization for IPv6:
2001:db8::/96 -> 2001:0db8:0000:0000:0000:0000:0000:0000/096 2001:db8:0:0:0:0:0:ffff -> 2001:0db8:0000:0000:0000:0000:0000:ffff/128
Queries
Combining the Data and Index patterns gives the ability to do basic querying. Queries are limited to the basic operators available in most databases, and at the moment do not expose in-array and not-in-array containers. A query may have the following operators:
gt
gte
lt
lte
eq
neq
or
and
not
Queries are permitted whether or not an index covers the query. If it the index does not cover the query, the query takes longer. For example, a query to find all Notable Events
authored by userA or userB might look like:
{ $or: [ { author: "userA" }, { author: "userB" } ] }
REST methods
The REST API for App KV Store implements the GET, POST, and DELETE methods described in this reference.
Defining KV Store lookups
For information on using the REST API to define or update a KV Store lookup, see data/transforms/lookups/
and data/transforms/lookups/{name}
.
storage/collections/config
https://<host>:<mPort>/servicesNS/{owner}/{app}/storage/collections/config
Authentication: Required
Description
Create or list collections.
Method summary
Method | Description | Formats |
---|---|---|
GET | List all collections. | XML, JSON |
POST | Create a new collection. | XML, JSON |
GET storage/collections/config
method detail
Request parameters
None
Response data keys
Name | Description |
---|---|
disabled | Collection state: true = disabled. false = [Default] enabled. |
profilingEnabled | Profiling status of slow-running operations: true = Profiling enabled. false = (Default) Profiling disabled. |
profilingThresholdMs | Threshold for logging slow-running operations, in milliseconds. Applies only if profilingEnabled = true. |
POST storage/collections/config
method detail
Request parameters
Name | Type | Default | Description |
---|---|---|---|
name required |
String | Collection name. | |
profilingEnabled | Boolean | A collections.conf file property. Affects profilingThresholdMs. Enable profiling of slow-running operations: true = Enable profiling. false = (Default) Disable profiling. | |
profilingThresholdMs | Number | Threshold for logging slow-running operations, in milliseconds. Applies only if profilingEnabled = true. This can have a performance impact on the collection. 0 = Log all slow-running operations. Default = 100. |
Response data keys
Name | Description |
---|---|
disabled | Collection state: true = disabled. false = [Default] enabled. |
profilingEnabled | Profiling status of slow-running operations, affecting profilingThresholdMs: true = Profiling enabled. false = (Default) Profiling disabled. |
profilingThresholdMs | Threshold for logging slow-running operations, in milliseconds. Applies only if profilingEnabled = true. |
Application usage
[ Top ]
storage/collections/config/{collection}
https://<host>:<mPort>/servicesNS/{owner}/{app}/storage/collections/config/{collection}
Authentication: Required
Description
Manage a specific collection.
Method summary
Method | Description | Formats |
---|---|---|
DELETE | Drop the {collection} collection. | XML, JSON |
GET | Get {collection} collection information. | XML, JSON |
POST | Update the {collection} collection. | XML, JSON |
DELETE storage/collections/config/{collection}
method detail
Request parameters
None
Response data keys
None
GET storage/collections/config/{collection}
method detail
Request parameters
None
Response data keys
Name | Description |
---|---|
disabled | Collection state: true = disabled. false = [Default] enabled. |
field.<fieldType> | Field type for field <fieldType>. One of: array (implicit) number bool string cidr (implicit) time |
index.<indexName> | Index definition named <indexName>. |
profilingEnabled | Profiling status of slow-running operations, affecting profilingThresholdMs: true = Profiling enabled. false = (Default) Profiling disabled. |
profilingThresholdMs | Threshold for logging slow-running operations, in milliseconds. Applies only if profilingEnabled = true. |
POST storage/collections/config/{collection}
method detail
Request parameters
Name | Type | Default | Description |
---|---|---|---|
field.<fieldType> | String | Field type for field <fieldType>. One of: array (implicit) number bool string cidr (implicit) time | |
index.<indexName> | String | Index definition named <indexName>. |
Response data keys
Name | Description |
---|---|
disabled | Collection state: true = disabled. false = [Default] enabled. |
field.<fieldType> | Field type for field <fieldType>. One of: array (implicit) number bool string cidr (implicit) time |
index.<indexName> | Index definition named <indexName>. |
profilingEnabled | Profiling status of slow-running operations, affecting profilingThresholdMs: true = Profiling enabled. false = (Default) Profiling disabled. |
profilingThresholdMs | Threshold for logging slow-running operations, in milliseconds. Applies only if profilingEnabled = true. |
Application usage
[ Top ]
storage/collections/data/{collection}
https://<host>:<mPort>/servicesNS/{owner}/{app}/storage/collections/data/{collection}
Authentication: Required
Description
Manage items of a collection.
Method summary
Method | Description | Formats |
---|---|---|
DELETE | Delete item(s) in the {collection} collection. | XML, JSON |
GET | Get items in the {collection} collection. | XML, JSON |
POST | Insert an item in the {collection} collection. | JSON |
DELETE storage/collections/data/{collection}
method detail
Request parameters
Name | Type | Default | Description |
---|---|---|---|
query | JSON object | Query JSON object. Conditional operators: $gt , $gte , $lt , $lte , and $ne Logical operators: $and , $or , and ,$not (invert conditional operators)Examples: query={"title":"Item"} (Select all documents with property title that has value Item )query={"price":{"$gt":5}} (Select all documents with price greater than 5 )
|
Response data keys
None
Application usage
The request deletes entries matching the query. If no query is provided, all entries in the collection are deleted.
GET storage/collections/data/{collection}
method detail
Request parameters
Name | Type | Default | Description |
---|---|---|---|
fields | String | Comma-separated list of fields to include (1 ) or exclude (0 ). A fields value cannot contain both include and exclude specifications except for exclusion of the _id field. Examples:fields=firstname,surname (Include only firstname , surname , and _id fields)fields=firstname,surname,_id:0 (Include only the firstname and surname fields.fields=address:0 (Include all fields except the address field)
| |
limit | Number | Maximum number of items to return. Example:limit=5 (Return five items)
| |
skip | Number | Number of items to skip from the start. Example:skip=10 (Skip first 10 items)
| |
sort | String | Sort order. Examples:sort=surname (Sort by surname , ascending)sort=surname,firstname (Sort by surname , ascending, after firstname , ascending)sort=surname:-1,firstname:1 (Sort by surname , descending, after firstname , ascendingsort=surname:1,first name (Sort by surname , ascending, after firstname , ascending
| |
query | JSON object | Query JSON object. Conditional operators: $gt , $gte , $lt , $lte , and $ne Logical operators: $and , $or , and ,$not (invert conditional operators)Examples: query={"title":"Item"} (Select all documents with property title that has value Item )query={"price":{"$gt":5}} (Select all documents with price greater than 5 )
|
Response data keys
JSON document. Example:
[ { <span style="color: #008000; font-weight: bold">"myKey"</span> : <span style="color: #BA2121">"abc"</span>, <span style="color: #008000; font-weight: bold">"_user"</span> : <span style="color: #BA2121">"nobody"</span>, <span style="color: #008000; font-weight: bold">"_key"</span> : <span style="color: #BA2121">"5410be5441ba15298e4624d1"</span> } ]
Application usage
Parameter order does not matter. Parameters are always applied in the order of sort, skip, and limit.
POST storage/collections/data/{collection}
method detail
Request parameters
JSON-formatted document.
Example:
{ "name": "A" }
Response data keys
JSON-formatted document key.
Example:
{ "_key": "530bc62fc9a6577fdf13651f" }
Application usage
The Content-Type header must be application/json.
The _id is autogenerated, if not manually specified.
[ Top ]
storage/collections/data/{collection}/{key}
https://<host>:<mPort>/servicesNS/{owner}/{app}/storage/collections/data/{collection}/{key}
Authentication: Required
Description
Manage a specific item of a collection.
Method summary
Method | Description | Formats |
---|---|---|
DELETE | Delete item {key} in {collection}. | XML, JSON |
GET | Get item in {collection} by {key} | XML, JSON |
POST | Update the {key} item in {collection}. | JSON |
DELETE storage/collections/data/{collection}/{key}
method detail
Request parameters
None
Response data keys
None
GET storage/collections/data/{collection}/{key}
method detail
Request parameters
None
Response data keys
JSON-formatted {key} document.
Example:
{"myKey" : "abc", "myOtherKey" : "uvwxyz", "_user" : "nobody", "_key" : "5410c8dc41ba15298e4624d5"}
POST storage/collections/data/{collection}/{key}
method detail
Request parameters
JSON-formatted document for {key}.
Example:
{ "myKey": "fizz"}
Response data keys
JSON-formatted {key}.
Example:
{"_key":"5410c8dc41ba15298e4624d5"}
Application usage
The header Content-Type must be application/json.
[ Top ]
storage/collections/data/{collection}/batch_save
https://<host>:<mPort>/servicesNS/storage/collections/data/{collection}/batch_save
Authentication: Required
Description
Perform multiple save operations.
Method summary
Method | Description | Formats |
---|---|---|
POST | Perform multiple save operations. | JSON |
POST storage/collections/data/{collection}/batch_save
method detail
Request parameters
JSON array of documents to save.
Response data keys
JSON-formatted list of keys added.
Example:
[ "5410c43241ba15298e4624d3", "5418c54e41ba152267763c01" ]
Application usage
Header Content-Type must be application/json.
If a document does not have an _key field or store does not have a matching document ID, an _key new document is created.
If an operation fails, the server stops batch processing and returns an HTTP status code other than 200.
[ Top ]
PREVIOUS Knowledge endpoint examples |
NEXT KV store endpoint examples |
This documentation applies to the following versions of Splunk® Enterprise: 6.2.0, 6.2.1, 6.2.2, 6.2.3, 6.2.4, 6.2.5, 6.2.6, 6.2.7, 6.2.8, 6.2.9, 6.2.10, 6.2.11, 6.2.12, 6.2.13, 6.2.14, 6.2.15, 6.3.0, 6.3.1, 6.3.2, 6.3.3, 6.3.4, 6.3.5, 6.3.6, 6.3.7, 6.3.8, 6.3.9, 6.3.10, 6.3.11, 6.3.12, 6.3.13, 6.3.14, 6.4.0, 6.4.1, 6.4.2, 6.4.3, 6.4.4, 6.4.5, 6.4.6, 6.4.7, 6.4.8, 6.4.9, 6.4.10, 6.4.11
Feedback submitted, thanks!