Manage data models

The Data Models management page is where you go to create data models and maintain some of their "higher order" aspects such as permissions and acceleration. On this page you can:

• Create a new data model - It's as easy as clicking a button.
• Set permissions - Data models are knowledge objects and as such are permissionable. You use permissions to determine who can see and update the data model.
• Enable data model acceleration - This can speed up Pivot performance for data models that cover large datasets.
• Clone data models - Useful for quick creation of new data models that are based on existing data models, or to copy data models into other apps.
• Upload and download data models - Download a data model (export it outside of Splunk). Upload an exported data model into a different Splunk implementation.
• Delete data models - Remove data models that are no longer useful.

In this topic we'll discuss these aspects of data model management. When you need to define the dataset hierarchies that make up a data model, you go to the Data Model Editor. See Design data models.

Navigating to the Data Models management page

The Data Models management page is essentially a listing page, similar to the Alerts, Reports, and Dashboards listing pages. It enables management of permissions and acceleration and also enables data model cloning and removal. It is different from the Select a Data Model page that you may see when you first enter Pivot (you'll only see it if you have more than one data model), as that page exists only to enable Pivot users to choose the data model they wish to use for pivot creation.

The Data Models management page lists all of the data models in your system in a paginated table. This table can be filtered by app, owner, and name. It can also display all data models that are visible to users of a selected app or just show those data models that were actually created within the app.

If you use Splunk Cloud Platform, or if you use Splunk Enterprise and have installed the Splunk Datasets Add-on, you may also see table datasets in the Data Models management page.

See About datasets for more information about table datasets.

There are two ways to get to the Data Models management page. You can use the Settings list, or you can get there through the Datasets listing page and Data Model Editor.

Through the Settings list

Navigate to Settings > Data Models.

Through the Datasets listing page

1. In the Search & Reporting app, open the Datasets listing page.
2. Locate a data model dataset.
3. (Optional) Click the name of the data model dataset to view it in the dataset viewing page.
4. Select Manage > Edit Data Model for that dataset.
5. On the Data Model Editor, click All Data Models to go to the Data Models management page.

Create a new data model

Prerequisites

You can only create data models if your permissions enable you to do so. Your role must have the ability to write to at least one app. If your role has insufficient permissions the New Data Model button will not appear.

See Enable roles to create data models.

Steps

1. Navigate to the Data Models management page.
2. Click New Data Model to create a new data model.
3. Enter the data model Title.
   The Title field can accept any character except asterisks. It can also accept blank spaces between characters.
   The data model ID field fills in as you enter the title. Do not update it. The data model ID must be a unique identifier for the data model. It can only contain letters, numbers, and underscores. Spaces between characters are also not allowed. After you click Create you cannot change the ID value.
4. (Optional) Enter the data model Description.
5. (Optional) Change the 'App value if you want the data model to belong to a different app context. App displays app context that you are in currently.
6. Click Create to open the new data model in the Data Model Editor, where you can begin adding and defining the datasets that make up the data model.

When you first enter the Data Model Editor for a new data model it will not have any datasets. To define the data model's first dataset, click Add Dataset and select a dataset type. For more information about dataset definition, see the following sections on adding field, search, transaction, and child datasets.

For all the details on the Data Model Editor and the work of creating data model datasets, see Design data models.

Enable roles to create data models

By default only users with the admin or power role can create data models. For other users, the ability to create a data model is tied to whether their roles have "write" access to an app. To grant another role write access to an app, follow these steps.

Steps

1. Click the App dropdown at the top of the page and select Manage Apps to go to the Apps page.
2. On the Apps page, find the app that you want to grant data model creation permissions for and click Permissions.
3. On the Permissions page for the app, select Write for the roles that should be able to create data models for the app.
4. Click Save to save your changes.

Giving roles the ability to create data models can have other implications.

See Disable or delete knowledge objects.

About data model permissions

Data models are knowledge objects, and as such the ability to view and edit them is governed by role-based permissions. When you first create a data model it is private to you, which means that no other user can view it on the Select a Data Model page or Data Models management page or update it in any way.

If you want to accelerate a data model, you need to share it first. You cannot accelerate private data models. See Enable data model acceleration.

Align data model permissions with those of related knowledge objects

When you share a data model the knowledge objects associated with that data model (such as lookups or field extractions) must have the same permissions. Otherwise, people may encounter errors when they use the data model.

For example, if your data model is shared to all users of the Search app but uses a lookup table and lookup definition that is only shared with users of the Search app that have the Admin role, everything will work fine for Admin role users, but all other users will get errors when they try to use the data model in Pivot. The solution is either to restrict the data model to Admin users or to share the lookup table and lookup definition to all users of the Search app.

Edit the permissions for a data model

Prerequisites

• Manage knowledge object permissions

Steps

1. Go to the Data Models management page.
2. Locate the data model that you want to edit permissions for. Use one of the following options.

   Option	Additional steps for this option
   Select Edit > Edit Permissions.	None
   Expand the row for the dataset.	Click Edit for permissions.

3. Edit the dataset permissions and click Save to save your changes.

This brings up the Edit Permissions dialog, which you can use to share private data models with others, and to determine the access levels that various roles have to the data models.

Manage data model acceleration

Accelerated data models can return search results faster than they ordinarily would. After you enable acceleration for a data model, you can inspect its metrics to ensure it is being accelerated correctly. If you determine that there are problems, you can rebuild the data summary for the data model.

Enable data model acceleration

After you enable acceleration for a data model, pivots, reports, and dashboard panels that use that data model can return results faster than they did before.

Data model acceleration builds a data summary for a data model at the indexer level. This summary can be made up of several smaller summaries distributed across your indexers.

If your Splunk deployment utilizes distributed search, you may find that you are accelerating the same or similar data models on separate search heads or search head clusters. If this is the case, and if you have edit access to datamodels.conf for your Splunk implementation, you can arrange to have those data models share the same data model acceleration summary. This practice reduces the amount of indexer space used up by data model acceleration summaries and cuts down on redundant summary creation and search effort.

For more information, see Share data model acceleration summaries among search head clusters.

After the data summary is built, searches that use accelerated data model datasets run against the summary rather than the full array of _raw data. This can speed up data model search completion times by a significant amount.

While data model acceleration is useful for speeding up searches on extremely large datasets, it has a few caveats.

• After you accelerate a data model, you cannot edit it. To make changes to an accelerated data model, you must disable its acceleration. Reaccelerating the data model can be resource-intensive, so it's best to avoid disabling acceleration if you can.
• Data model acceleration is applied only to root event datasets, root search datasets that restrict their command usage to streaming commands, and their child datasets. The Splunk platform cannot apply acceleration to dataset hierarchies based on root transaction datasets or root search datasets that use nonstreaming commands. Searches that use those unaccelerated datasets fall back to _raw data.
• Data model acceleration is most efficient if the root event datasets or root search datasets include the indexes to be searched in their initial constraint search. Otherwise, all available indexes for the data model are searched.
• Search filters cannot be applied to accelerated data model datasets. You cannot apply either role-based or user-based search filters to an accelerated data model.

Prerequisites

• A role with the accelerate_datamodel capability, such as the admin role. Data model acceleration can be resource-intensive, so it should be used conservatively by a limited number of users.
• The data model must be shared with other users before you can accelerate it, and any knowledge objects that the data model is dependent on must be shared as well. See About data model permissions.
• See Accelerate data models.
• See Command types in the Search Reference for more information about streaming, generating, and transforming commands.
• See Specify time modifiers in your search in the Search Manual or learn about setting fixed UNIX time dates if you intend to enter a Custom Summary Range.

Steps

1. In Splunk Web, go to to the Data Models management page.
2. Find the data model you want to accelerate and open its acceleration controls. Use one of the following options:

   Option	Additional steps for this option
   Navigate to the Data Models management page.	Find the model you want to accelerate and select Edit > Edit Acceleration.
   Navigate to the Data Models management page.	Expand the row of the data model you want to accelerate and click Add for ACCELERATION.
   Open the Data Model Editor for a data model.	Select Edit > Edit Acceleration.

3. Select Accelerate to enable acceleration for the data model.
4. Select a Summary Range of 1 Day, 7 Days, 1 Month, 3 Months, 1 Year, All Time, or Custom depending on the range of time over which you plan to run searches that use the accelerated datasets within the data model. For example, if you only plan to run searches with this data model over periods of time within the last seven days, choose 7 Days.
   
   Select Custom to provide a custom earliest time range. You can use relative time notation, or you can provide a fixed date in Unix epoch time format.
   
   Smaller time ranges result in smaller summaries that require less time to build and take up less space on disc.
5. (Optional)Open Advanced Settings to access advanced data model acceleration settings. Change these settings only if you are experiencing summary creation issues. For more information about advanced data model acceleration settings, see Accelerate data models.
6. Select Save.
   
   After your data model is accelerated, the icon for the model on the Data Models management page is yellow instead of gray.

Inspect data model acceleration metrics

After a data model is accelerated, you can find information about the model's acceleration on the Data Models management page. Expand the row for the accelerated data model and review the information that appears under ACCELERATION.

Field	Description
Status	Tells you whether the acceleration summary for the data model is complete. If it is in building status it will tell you what percentage of the summary is complete. Data model summaries can constantly update with new data. Just because a summary is complete now doesn't mean it won't be building later.
Access Count	Tells you how many times the data model summary has been accessed since it was created, and when the last access time was. This metric can help you determine which data models are infrequently used. Because data model acceleration uses system resources, you should restrict acceleration to data models that are accessed frequently.
Size on Disk	Shows you how much space the data model's acceleration summary takes up in terms of storage. You can use this metric along with the Access Count to determine which summaries are an unnecessary load on your system and ought to be deleted. If the acceleration summary for your data model is taking up a large amount of space on disk, you might also consider reducing its summary range.
Summary Range	The range of the data model, in seconds, always relative to the present moment. You set this range up when you define acceleration for the data model.
Buckets	The number of index buckets spanned by the data model acceleration summary.
Updated	Tells you when the summary was last updated with the results of a summarization search.

You can optionally expand Detailed Acceleration Information to see various kinds of runtime statistics, both overall and for the last run of the acceleration summarization search. Summarization searches should take a uniform amount of time to complete. If the overall runtime statistics indicate that there is a lot of variance in summarization runtimes, the environment might be unhealthy or the system might be overloaded.

Field	Description
SID	The search ID of the last data model acceleration summarization search job for this data model.
Start Time	The start time of the last data model acceleration summarization search job for this data model.
Run Time	The run time of the last data model acceleration summarization search job for this data model.
Average	The average run time of the search jobs that create the acceleration summary for this data model.
p50	The 50th percentile of summarization search runtimes for the data model. 50 percent of the summarization searches for this data model had runtimes that were less than this value.
p90	The 90th percentile of summarization search runtimes for the data model. 90 percent of the summarization searches for this data model had runtimes that were less than this value.

Finally, you can optionally expand Configuration Settings to review the configuration settings for this data model. You can edit some of these settings by selecting Edit and changing the Advanced Settings. Other settings, such as the hunk.dfs_block_size, can only be changed by editing the stanza for the data model in datamodels.conf.

Rebuild a summary for an accelerated data model

You may want to rebuild the summary for your data model if you suspect there has been data loss due to a system crash or similar mishap. When you rebuild your summary, Splunk software deletes the entire acceleration summary for this data model and rebuilds it. This can take a long time for larger summaries.

The Splunk platform automatically rebuilds summaries when you disable and then reenable acceleration for a summary. You might disable and reenable acceleration for a data model when you edit the data model, because the data model cannot be edited when it is in an accelerated state.

Prerequisites

• An accelerated data model.

Steps

1. In Splunk Web, go to the Data Models management page.
2. Find the accelerated data model that needs to have its summary rebuilt, and expand its row.
3. Click Rebuild. The summary will begin rebuilding.
4. (Optional) Check the Status of the summary to find out when it is complete.

Update summary metrics for an accelerated data model

Data model acceleration metrics are updated on a regular interval. If you do not want to wait for a scheduled update, you can get the metrics updated right away by clicking the Update button.

Prerequisites

• An accelerated data model.

Steps

1. In Splunk Web, go to the Data Models management page.
2. Expand the row of an accelerated data model to see its acceleration metrics.
3. Click Update to have the Splunk platform update the metrics it displays for the data model.

Edit the advanced data model acceleration settings

If you run into issues with summary creation for a data model, you may need to adjust its advanced data model acceleration settings. Click Edit to open the Edit Acceleration dialog and update the data model acceleration settings.

Clone a data model

Data model cloning is a way to quickly create a data model that is based on an existing data model. You can then edit it so it focuses on a different overall dataset or has a different dataset structure that divides up the dataset in a different way than the original.

Steps

1. Use one of the following options.

   Option	Additional steps for this option
   Go to the Data Models management page.	Locate the data model that you want to clone and select Edit > Clone.
   Open the Data Model Editor for the data model that you want to clone.	Select Edit > Clone.

2. Enter a unique name for the cloned data model in New Title.
3. (Optional) Provide a Description for the new data model.
4. (Optional) If your permissions allow it, select Clone to give the cloned data model the same permissions as the data model it is cloned from.
5. Click Clone to create the data model clone.

You can edit the cloned data model in the Data Model management page, and the Data Model Editor, as described in Design data models.

Upload and download data models

You can use the download/upload functionality to export a data model from one Splunk deployment and upload it into another Splunk deployment. You can use this feature to back up important data models, or to collaborate on data models with other Splunk users by emailing them to those users. You might also use it to move data models between staging and production instances of Splunk.

You can manually move data model JSON files between Splunk deployments, but this is an unsupported procedure with many opportunities for error.

See Manual data model management.

Download a data model

Download a data model from the Data Model Editor. You can only download one data model at a time.

Steps

1. Open a data model in the Data Model Editor.
2. Click the Download button at the top right.
   
   Splunk will download the JSON file for the data model to your designated download directory. If you haven't designated this directory, you may see a dialog that asks you to identify the directory you want to save the file to.

The name of the downloaded JSON file will be the same as the data model's ID. You provide the ID only once, when you first create the data model. Unlike the data model Title, once the ID is saved with the creation of the model, you can't change it.

You can see the ID for an existing data model when you view the model in the Data Model Editor. The ID appears near the top left corner of the Editor, under the model's title.

When you upload the data model you have an opportunity to give it a new ID that is different from the ID of the original data model.

Upload a data model

Upload a data model from the Data Models management page. You can only upload one data model at a time.

Splunk software validates any file that you try to upload. It cannot upload files that contain anything other than valid JSON data model code.

Steps

1. Navigate to the Data Models management page.
2. Click Upload Data Model.
3. Identify the JSON File that you want to upload.
   The ID field populates with the original ID of the data model.
4. (Optional) Change the data model ID to a new, unique value.
   Keep in mind that once you save the data model file to your system you will not be able to change this ID. You can still edit the data model title after you save it to your system.
5. Provide the name of the App that the data model belongs to.
6. (Optional) If your capabilities allow it, change the uploaded data model permissions from Private to Shared in App.
   
   • Shared in App indicates that the data model is shared with all users of the App.
     
   • If you select Shared in App you can also enable acceleration for the data model by selecting Accelerate and choosing a Summary Range.
7. Click Upload to upload the data model.
   The uploaded data model appears in the Data Model management page listing if it passes validation.

See About data model permissions.

See Enable data model acceleration.

Delete a data model

You can delete a data model from the Data Model Editor or the Data Models management page.

If your role has write access to your current app context you should be able to delete data models that belong to that app. For more information about this see Enable roles to create data models.

You cannot use Splunk Web to remove default data models that were delivered with Splunk software. Only data models that exist in an app's local directory are eligible for deletion.

See Disable or delete knowledge objects.

Delete a data model

1. In the Search & Reporting app, click Datasets to open the Datasets listing page.
2. Locate a data model dataset that belongs to the dataset that you want to delete.
3. Select Manage > Edit Dataset.
4. Delete the data model.

Manual data model management

Splunk does not recommend that you manage data models manually by hand-moving their files or hand-coding data model files. You should create and edit data models in Splunk Web whenever possible. When you edit models in Splunk Web the Data Model Editor validates your changes. The Data Model Editor cannot validate changes in models created or edited by hand.

Data models are stored on disk as JSON files. They have associated configs in datamodels.conf and metadata in local.meta (for models that you create) and default.meta (for models delivered with the product).

Models that you create are stored in <yourapp>/local/data/models, while models delivered with the product can be found in <yourapp>/default/data/models.

You can manually move model files between Splunk implementations but it's far easier to use the Data Model Download/Upload feature in Splunk Web (described above). If you absolutely must move model files manually, take care to move their datamodels.conf stanzas and local.meta metadata when you do so.

The same goes for deleting data models. In general it's best to do it through Splunk Web so the appropriate cleanup is carried out.