Manage data models
- Enable roles to create data models
- To enable data model acceleration
- Inspect data model acceleration metrics
- Download a data model
- Upload a data model
Manage data models
The Data Models management page is where you go to create data models and maintain some of their "higher order" aspects such as permissions and acceleration. On this page you can:
- Create a new data model - It's as easy as clicking a button.
- Set permissions - Data models are knowledge objects and as such are permissionable. You use permissions to determine who can see and update the data model.
- Enable data model acceleration - This can speed up Pivot performance for data models that cover large datasets.
- Clone data models - Useful for quick creation of new data models that are based on existing data models, or to copy data models into other apps.
- Upload and download data models - Download a data model (export it outside of Splunk). Upload an exported data model into a different Splunk implementation.
- Delete data models - Remove data models that are no longer useful.
In this topic we'll discuss these aspects of data model management. When you need to define the object hierarchies that make up a data model, you go to the Data Model Editor. For more information, "Design data models and objects," in this manual.
The Data Models management page is essentially a listing page, similar to the Alerts, Reports, and Dashboards listing pages. It enables management of permissions and acceleration and also enables data model cloning and removal. It's different from the Select a Data Model page that you may see when you first enter Pivot (you'll only see it if you have more than one data model), as that page exists only to enable Pivot users to choose the data model they wish to use for pivot creation.
The Data Models management page lists all of the data models in your system in a paginated table. This table can be filtered by app, owner, and name. It can also display all data models that are visible to users of a selected app or just show those data models that were actually created within the app.
Splunk Enterprise enables you to navigate to the Data Models management page via a few different avenues:
- You can access the page from anywhere in Splunk Web through the Settings list. Just navigate to Settings > Data Models.
- From the Data Models listing page in Pivot, click the Manage Data Models button.
- From the Data Model Editor, click Back To Models.
Create a new data model
You create data models by navigating to the Data Models management page (see above for instructions) and clicking New Data Model.
Note: You can only create data models if your role's permissions enable you to do so (your role must have the ability to write to at least one app). If your role has insufficient permissions the New Data Model button will not appear. For more information see the subtopic "Enable roles to create data models," below.
When you click New Data Model, Splunk Enterprise displays the Create New Data Model dialog. Enter the data model Title and optional Description.
The Title field can accept any character except asterisks. It can also accept blank spaces between characters. It's what you'll see on the Select a Data Model page and the Data Models management page, and elsewhere in Splunk Enterprise where the data model name is displayed.
The data model ID field will fill in as you enter the title; we advise that you do not update it. The data model ID must be a unique identifier for the data model. It can only contain letters, numbers, and underscores. Spaces between characters are also not allowed. Once you click Create you can't change the ID value.
App will display the app context that you are in currently. If you want the data model to belong to a different app, change the App value.
Click Create to open the new data model in the Data Model Editor, where you can begin adding and defining the objects that make up the data model.
When you first enter the Data Model Editor for a new data model it will not have any objects. To define the data model's first object, click Add Object and select an object type. For more information about object definition, see the following sections on adding field, search, transaction, and child objects.
For all the details on the Data Model Editor and the work of creating data model objects, see "Design data models and objects," in this manual.
Enable roles to create data models
By default only users with the admin or power role can create data models. For other users, the ability to create a data model is tied to whether their roles have "write" access to an app. To grant another role write access to an app, follow these steps:
1. Click the App dropdown at the top of the page and select Manage Apps to go to the Apps page.
2. On the Apps page, find the app that you want to grant data model creation permissions for and click Permissions.
3. On the Permissions page for the app, select Write for the roles that should be able to create data models for the app.
4. Click Save to save your changes.
Note: Giving roles the ability to create data models can have other implications. See "Disable or delete knowledge objects" in this manual for more information.
About data model permissions
Data models are knowledge objects, and as such the ability to view and edit them is governed by role-based permissions. When you first create a data model it is private to you, which means that no other user can view it on the Select a Data Model page or Data Models management page or update it in any way.
To edit the permissions for a data model, go to the Data Models management page, locate the data model and either:
- Click Edit and select Edit Permissions.
- Expand the row for the data model in question and click Edit for permissions.
This brings up the Edit Permissions dialog, which you can use to share private data models with others, and to determine the access levels that various roles have to the data models.
For more information about setting permissions for data models see "Manage knowledge object permissions" in this manual. By default any role can create a data model, but any data models those roles create will be private until a user with an admin or power role shares them. Only users with an admin or power role can create and share a data model.
Important: When you share a data model the knowledge objects associated with that data model (such as lookups or field extractions) must have the same permissions. Otherwise you run the risk of running into errors when other people try to use the data model.
For example, if your data model is shared to all users of the Search app but uses a lookup table that is only shared with users that have the Admin role, everything will work fine for Admin role users, but all other users will get errors that say things like "the lookup table does not exist" when they try to use the data model in Pivot. The solution is either to restrict the data model to Admin users or to share the lookup to all users of the Search app.
You'll also run into problems if your data model is private and the related lookup tables and lookup definitions are private, and then you decide to accelerate the data model. To accelerate a data model you must share it. If you do not share the related lookup tables and lookup definition in exactly the same way, your users will see "the lookup table does not exist" messages.
Enable data model acceleration
Data model acceleration enables you to speed up the dataset represented by a data model for reporting purposes. After a data model is accelerated, pivots, reports, and dashboard panels that use that data model should return results faster than they did before.
Data model acceleration is powered by the high performance analytics store. With the power of the high performance analytics store, data model acceleration builds a data summary for a data model at the index level (this summary can in fact be made up of several smaller summaries, distributed across your indexers). After the summary is completely built, pivots that use accelerated data model objects will run against the summary rather than the full array of
_raw data when possible. This can speed up pivot result return time by a significant amount.
While data model acceleration is useful for speeding up extremely large datasets, it comes with a few important caveats:
- By default, only users with admin permissions can accelerate data models. Data model acceleration can be resource-intensive, so it should be used conservatively by a limited number of Splunk Enterprise users. The ability to accelerate a data model is tied to the
- Data models that are private cannot be accelerated. You must share a data model with the users of an app to make it eligible for acceleration. When you do this, you need to share related knowledge objects (such as lookup tables and lookup definitions that your lookup attributes are dependent upon) as well, in exactly the same way. See "About data model permissions," above, for more information.
- Once a data model is accelerated, it can no longer be edited. You can't change an accelerated data model in any way until its acceleration is disabled. Reaccelerating the data model can also be resource-intensive so it's best to avoid disabling acceleration if you can.
- Data model acceleration only affects the first event object hierarchy in a data model. Additional event object hierarchies and object hierarchies based on root search and root transaction objects will not be accelerated.
- Data model acceleration is most efficient if the root event object being accelerated includes in its initial constraint search the index(es) that Splunk Enterprise should search over. Otherwise Splunk Enterprise will search over all available indexes for the data model, which could lead it to waste time accelerating unnecessary data.
It's also important to understand that Splunk Enterprise does not apply data model acceleration to all objects in a data model. When you accelerate a data model, Splunk Enterprise accelerates only the first base event object and its child objects. All other objects are not accelerated; pivots that use them will fall back to
In other words:
|Data Model Object Type/Description||Will be accelerated when |
acceleration is enabled?
|Topmost base event object||YES|
|Child of the topmost base event object||YES|
|Base event object that appears below the topmost base event object||NO|
|Child of a non-topmost base event object||NO|
|Base search object||NO|
|Base transaction object||NO|
With that in mind, if you are building a data model that you intend to accelerate, you may want to design it so that the first base event object spawns the bulk of the objects that you want to accelerate. If you find a need to add a secondary base event object to the data model, you may want to consider giving that secondary base event object its own data model so it can be accelerated independently of the first base event object.
For details about data model acceleration, including an explanation of what's happening behind the scenes and a discussion of "ad hoc" data model acceleration, see "Accelerate data models," in this manual.
To enable data model acceleration
If your permissions are sufficient to accelerate a data model, follow these steps:
1. Navigate to the Data Models management page.
2. Find the data model you want to accelerate and either click Edit and select Edit Acceleration OR expand the data model's row and click Add for ACCELERATION.
3. The Edit Acceleration dialog appears. Select the Accelerate checkbox to enable acceleration for the data model.
4. The Summary Range field appears. Select from 1 Day, 7 Days, 1 Month, 3 Months, 1 Year, or All Time depending on the range of time over which you plan to run pivots that use the accelerated objects within the data model. For example, if you only plan to run pivots over periods of time within the last seven days, choose 7 Days.
Note: If you require a different summary range than the ones supplied by the Summary Range field, you can configure it for your data model in
5. Click Save to save your acceleration settings. Once your data model is accelerated, the "lightning bolt" symbol for the model on the Data Models management page will be lit up with a yellow color.
Inspect data model acceleration metrics
After a data model is accelerated, you can find detail information about the model's acceleration on the Data Models management page. Just expand the row for the accelerated data model and review the information that appears under ACCELERATION.
- Status tells you whether the acceleration summary for the data model is complete. If it is in Building status it will tell you what percentage of the summary is complete. Keep in mind that many data model summaries are constantly updating with new data; just because a summary is "complete" now doesn't mean it won't be "building" later.
- Access Count tells you how many times the data model summary has been accessed since it was created, and when the last access time was. This can be useful if you're trying to determine which data models are not being used frequently. Because data model acceleration uses system resources you may not want to accelerate data models that aren't accessed on a regular basis.
- Size on Disk hows you how much space the data model's acceleration summary takes up in terms of storage. You can use this metric along with the Access Count to determine which summaries are an unnecessary load on your system and ought to be deleted. If the acceleration summary for your data model is taking up a large amount of space on disk, you might also consider reducing its summary range.
- Summary Range presents the range of the data model, in seconds, always relative to the present moment. You set this range up when you define acceleration for the data model.
- Buckets displays the number of index buckets spanned by the data model acceleration summary.
Click Rebuild to make Splunk Enterprise rebuild the summary from scratch. You may want to do this in situations where you suspect there has been data loss due to a system crash or similar mishap. Splunk Enterprise automatically rebuilds summaries when you disable and then reenable acceleration for a summary (to edit the data model, for example).
Click Update to refresh the acceleration summary detail information.
Click Edit to open the Edit Acceleration dialog and change the Summary Range or disable acceleration for the data model altogether.
Clone a data model
Data model cloning is a way to quickly create a data model that is based on an existing data model. You can then edit it so it focuses on a different overall dataset or has a different object structure that divides up the dataset in a different way than the original. To clone a data model go to the Data Model management page, click Edit for the data model that you want to clone, and select Clone. Splunk Enterprise will create a new data model that is identical to the original. You will have to give the cloned data model a unique name.
Note: You can also clone a data model from the Data Model Editor. Simply click Edit and select Clone.
You can edit the cloned data model with the Data Model management page (as described in this topic) and the Data Model Editor (as described in "Design data models and objects," in this manual).
Upload and download data models
Splunk's data model download/upload functionality lets you use the Splunk Web interface to export a data model out of Splunk and then upload that data model into another Splunk implementation. You can use this feature to back up important data models, or to collaborate on data models with other Splunk users by emailing them to those users. You might also use it to move data models between staging and production instances of Splunk.
Note: You can manually move data model JSON files between Splunk implementations, but this is an unsupported procedure with many opportunities for error. For more information, see "Manual data model management" at the bottom of this topic.
Download a data model
Download a data model from the Data Model Editor. You can only download one data model at a time.
To download a data model, open the data model in the Data Model Editor and click the Download button at the top right. Splunk will download the JSON file for the data model to your designated download directory. If you haven't designated this directory, you may see a dialog that asks you to identify the directory you want to save the file to.
The name of the downloaded JSON file will be the same as the data model's ID. You provide the ID only once, when you first create the data model. Unlike the data model Title, once the ID is saved with the creation of the model, you can't change it.
You can see the ID for an existing data model when you view the model in the Data Model Editor. The ID appears near the top left corner of the Editor, under the model's title.
When you upload the data model you will have an opportunity to give it a new ID that is different from the ID of the original data model.
Upload a data model
Upload a data model from the Data Models management page. You can only upload one data model at a time.
Note: Splunk will validate any file that you try to upload to your Splunk implementation. It will not upload a file if it contains anything other than valid JSON data model code.
To upload a data model to your Splunk implementation, click Upload Data Model (near the top right corner of the page) to open the Upload New Data Model dialog box. Start by identifying the JSON File that you want to upload.
The ID field will populate with the original ID of the data model. You can change this ID if you wish. Keep in mind that once you save the data model file to your system you will not be able to change this ID (but you can edit the data model title).
Provide the name of the App that the data model belongs to and identify whether the data model is Private or Shared in App (meaning that it is shared with all other users of the app).
For more information about data model permissions, see "About data model permissions," above.
If you select Shared in App you can also enable acceleration for the data model by selecting Accelerate and choosing a Summary Range.
For more information about enabling data model acceleration, see "Enable data model acceleration," above.
Delete a data model
You can delete a data model from the Data Model management page or the Data Model Editor. Just click Edit and select Delete.
Note: If your role grants you the ability to create data models, it should grant you the ability to delete them as well. For more information about this see "Enable roles to create data models," above.
Manual data model management
Splunk does not recommend that you manage data models manually by hand-moving their files or hand-coding data model files. You should create and edit data models in Splunk Web whenever possible. When you edit models in Splunk Web the Data Model Editor validates your changes; this won't happen for models created or edited by hand.
Data models are stored on disk as JSON files, and they have associated configs in
datamodels.conf and metadata in
local.meta (for models that you create) and
default.meta (for models delivered with the product).
Models that you create are stored in
<yourapp>/local/data/models while models delivered with the product can be found in
You can move model files between Splunk implementations manually but it's far easier to use the Data Model Download/Upload feature in Splunk Web (described above). If you absolutely must move model files manually, take care to move their
datamodels.conf stanzas and
local.meta metadata when you do so.
The same goes for deleting data models; in general it's best to do it via Splunk Web so all the appropriate cleanup is carried out.