Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

Download topic as PDF

Manage data models

The Data Models management page is where you go to create data models and maintain some of their "higher order" aspects such as permissions and acceleration. On this page you can:

  • Create a new data model - It's as easy as clicking a button.
  • Set permissions - Data models are knowledge objects and as such are permissionable. You use permissions to determine who can see and update the data model.
  • Enable data model acceleration - This can speed up Pivot performance for data models that cover large datasets.
  • Clone data models - Useful for quick creation of new data models that are based on existing data models, or to copy data models into other apps.
  • Upload and download data models - Download a data model (export it outside of Splunk). Upload an exported data model into a different Splunk implementation.
  • Delete data models - Remove data models that are no longer useful.

In this topic we'll discuss these aspects of data model management. When you need to define the dataset hierarchies that make up a data model, you go to the Data Model Editor. For more information, Design data model datasets.

Navigating to the Data Models management page

The Data Models management page is essentially a listing page, similar to the Alerts, Reports, and Dashboards listing pages. It enables management of permissions and acceleration and also enables data model cloning and removal. It is different from the Select a Data Model page that you may see when you first enter Pivot (you'll only see it if you have more than one data model), as that page exists only to enable Pivot users to choose the data model they wish to use for pivot creation.

The Data Models management page lists all of the data models in your system in a paginated table. This table can be filtered by app, owner, and name. It can also display all data models that are visible to users of a selected app or just show those data models that were actually created within the app.

If you use Splunk Cloud, or if you use Splunk Enterprise and have installed the Splunk Datasets Add-on, you may also see table datasets in the Data Models management page.

See About datasets for more information about table datasets.

There are two ways to get to the Data Models management page. You can use the Settings list, or you can get there through the Datasets listing page and Data Model Editor.

Through the Settings list

Navigate to Settings > Data Models.

Through the Datasets listing page

  1. In the Search & Reporting app, open the Datasets listing page.
  2. Locate a data model dataset.
  3. (Optional) Click the name of the data model dataset to view it in the dataset viewing page.
  4. Select Manage > Edit Data Model for that dataset.
  5. On the Data Model Editor, click All Data Models to go to the Data Models management page.

Create a new data model

Prerequisites

You can only create data models if your permissions enable you to do so. Your role must have the ability to write to at least one app. If your role has insufficient permissions the New Data Model button will not appear.

See Enable roles to create data models.

Steps

  1. Navigate to the Data Models management page.
  2. Click New Data Model to create a new data model.
  3. Type the data model Title.
    The Title field can accept any character except asterisks. It can also accept blank spaces between characters.
    The data model ID field fills in as you enter the title. Do not update it. The data model ID must be a unique identifier for the data model. It can only contain letters, numbers, and underscores. Spaces between characters are also not allowed. After you click Create you cannot change the ID value.
  4. (Optional) Type the data model Description.
  5. (Optional) Change the App value if you want the data model to belong to a different app context. App displays the app context that you are in currently.
  6. Click Create to open the new data model in the Data Model Editor, where you can begin adding and defining the datasets that make up the data model.

Bubbles dm createnew mod.png

When you first enter the Data Model Editor for a new data model it does not have any datasets. To define the data model's first dataset, click Add Dataset and select a dataset type. For more information about dataset definition, see the following sections on adding field, search, transaction, and child datasets.

For all the details on the Data Model Editor and the work of creating data model datasets, see Design data model datasets.

Enable roles to create data models

By default only users with the admin or power role can create data models. For other users, the ability to create a data model is tied to whether their roles have "write" access to an app. To grant another role write access to an app, follow these steps.

Steps

  1. Click the App dropdown at the top of the page and select Manage Apps to go to the Apps page.
  2. On the Apps page, find the app that you want to grant data model creation permissions for and click Permissions.
  3. On the Permissions page for the app, select Write for the roles that should be able to create data models for the app.
  4. Click Save to save your changes.

Giving roles the ability to create data models can have other implications.

See Disable or delete knowledge objects.

About data model permissions

Data models are knowledge objects, and the ability to view and edit them is determined by role-based permissions. When you first create a data model it is private to you, which means that no other user can view it on the Select a Data Model page or Data Models management page or update it in any way.

If you want to accelerate a data model, you need to share it first. You cannot accelerate private data models. See Enable data model acceleration.

Align data model permissions with those of related knowledge objects

When you share a data model the knowledge objects associated with that data model (such as lookups or field extractions) must have the same permissions. Otherwise, people may encounter errors when they use the data model.

For example, if your data model is shared to all users of the Search app but uses a lookup table and lookup definition that is only shared with users that have the Admin role, everything will work fine for Admin role users, but all other users will get errors when they try to use the data model in Pivot. The solution is either to restrict the data model to Admin users or to share the lookup table and lookup definition to all users of the Search app.

Edit the permissions for a data model

Prerequisites

Steps

  1. Go to the Data Models management page.
  2. Locate the data model that you want to edit permissions for. Use one of the following options.
    Option Additional steps for this option
    Select Edit > Edit Permissions. None
    Expand the row for the dataset. Click Edit for permissions.
  3. Edit the dataset permissions and click Save to save your changes.

This brings up the Edit Permissions dialog, which you can use to share private data models with others, and to determine the access levels that various roles have to the data models.

Enable data model acceleration

After you enable acceleration for a data model, pivots, reports, and dashboard panels that use that data model can return results faster than they did before.

Data model acceleration is powered by the high performance analytics store. With the power of the high performance analytics store, data model acceleration builds a data summary for a data model at the index level. This summary can be made up of several smaller summaries, distributed across your indexers.

After the summary is completely built, pivots that use accelerated data model datasets run against the summary rather than the full array of _raw data when possible. This can speed up pivot result return time by a significant amount.

While data model acceleration is useful for speeding up extremely large datasets, it comes with a few important caveats.

  • By default, only users with admin permissions can accelerate data models. Data model acceleration can be resource-intensive, so it should be used conservatively by a limited number of Splunk users. The ability to accelerate a data model is tied to the accelerate_datamodel capability.
  • You cannot enable acceleration for private data models. You must share a data model with the users of an app to make it eligible for acceleration. When you do this, you need to share related knowledge objects (such as lookup tables and lookup definitions that your lookup fields are dependent upon) as well, in exactly the same way. See "About data model permissions," above, for more information.
  • Once you accelerate a data model, you cannot edit it. If you need to make changes to an accelerated data model, you need to disable its acceleration. Reaccelerating the data model can be resource-intensive so it's best to avoid disabling acceleration if you can.
  • Data model acceleration can only be applied to root event datasets, root search datasets that restrict their command usage to streaming commands, and their child datasets. Dataset hierarchies based on root transaction datasets or root search datasets that use non-streaming commands cannot be accelerated. Pivots that use those unaccelerated datasets fall back to _raw data.
  • Data model acceleration is most efficient if the root event datasets or root search datasets being accelerated include the index(es) to be searched in their initial constraint search. Otherwise all available indexes for the data model are searched, which can waste time accelerating unnecessary data.

See Accelerate data models in this manual.

See Command types in the Search Reference for more information about streaming, generating, and transforming commands.

To enable data model acceleration

If your permissions are sufficient to accelerate a data model, follow these steps:

  1. Navigate to the Data Models management page.
  2. Find the data model you want to accelerate and open its acceleration controls. Use one of the following options.
    Option Additional steps for this option
    Select Edit > Edit Acceleration. None
    Expand the row for the data model. Click Add for ACCELERATION.
  3. Select Accelerate to enable acceleration for the data model.
    6.0 dm edit acceleration dialog.png
  4. Select a Summary Range of 1 Day, 7 Days, 1 Month, 3 Months, 1 Year, or All Time, depending on the range of time over which you plan to run pivots that use the accelerated datasets within the data model.

    For example, if you only plan to run pivots over periods of time within the last seven days, choose 7 Days.

    If you require a different summary range than the ones supplied by the Summary Range field, you can configure it for your data model in datamodels.conf.
  5. Click Save to save your acceleration settings.

    Once your data model is accelerated, the "lightning bolt" symbol for the model on the Data Models management page will be lit up with a yellow color.
    6.0 dm acceleration lightning bolt.png

Inspect data model acceleration metrics

After a data model is accelerated, you can find detail information about the model's acceleration on the Data Models management page. Just expand the row for the accelerated data model and review the information that appears under ACCELERATION.

6.0 dm acceleration metrics.png

  • Status tells you whether the acceleration summary for the data model is complete. If it is in Building status it will tell you what percentage of the summary is complete. Keep in mind that many data model summaries are constantly updating with new data; just because a summary is "complete" now doesn't mean it won't be "building" later.
  • Access Count tells you how many times the data model summary has been accessed since it was created, and when the last access time was. This can be useful if you're trying to determine which data models are not being used frequently. Because data model acceleration uses system resources you may not want to accelerate data models that aren't accessed on a regular basis.
  • Size on Disk hows you how much space the data model's acceleration summary takes up in terms of storage. You can use this metric along with the Access Count to determine which summaries are an unnecessary load on your system and ought to be deleted. If the acceleration summary for your data model is taking up a large amount of space on disk, you might also consider reducing its summary range.
  • Summary Range presents the range of the data model, in seconds, always relative to the present moment. You set this range up when you define acceleration for the data model.
  • Buckets displays the number of index buckets spanned by the data model acceleration summary.

Click Rebuild to rebuild the summary from scratch. You may want to do this in situations where you suspect there has been data loss due to a system crash or similar mishap. Splunk Enterprise automatically rebuilds summaries when you disable and then reenable acceleration for a summary (to edit the data model, for example).

Click Update to refresh the acceleration summary detail information.

Click Edit to open the Edit Acceleration dialog and change the Summary Range or disable acceleration for the data model altogether.

Clone a data model

Data model cloning is a way to quickly create a data model that is based on an existing data model. You can then edit it so it focuses on a different overall dataset or has a different dataset structure that divides up the dataset in a different way than the original.

Steps

  1. Use one of the following options.
    Option Additional steps for this option
    Go to the Data Models management page. Locate the data model that you want to clone and select Edit > Clone.
    Open the Data Model Editor for the data model that you want to clone. Select Edit > Clone.
  2. Enter a unique name for the cloned data model in New Title.
  3. (Optional) Provide a Description for the new data model.
  4. (Optional) If your permissions allow it, select Clone to give the cloned data model the same permissions as the data model it is cloned from.
  5. Click Clone to create the data model clone.

You can edit the cloned data model with the Data Model management page, as described in this topic, and the Data Model Editor, as described in Design data model datasets.

Upload and download data models

You can use the download and upload functionality to export a data model from one Splunk deployment and upload it into another Splunk deployment. You can use this feature to back up important data models or to collaborate on data models with other Splunk users by emailing them to those users. You might also use it to move data models between staging and production instances of Splunk.

You can manually move data model JSON files between Splunk deployments, but this is an unsupported feature.

See Manual data model management.

Download a data model

Download a data model from the Data Model Editor. You can only download one data model at a time.

Steps

  1. Open a data model in the Data Model Editor.
  2. Click the Download button at the top right.

    Splunk will download the JSON file for the data model to your designated download directory. If you haven't designated this directory, you may see a dialog that asks you to identify the directory you want to save the file to.

Cupk dm download button.png

The name of the downloaded JSON file will be the same as the data model's ID. You provide the ID only once, when you first create the data model. Unlike the data model Title, once the ID is saved with the creation of the model, you can't change it.

You can see the ID for an existing data model when you view the model in the Data Model Editor. The ID appears near the top left corner of the Editor, under the model's title.

When you upload the data model you have an opportunity to give it a new ID that is different from the ID of the original data model.

Upload a data model

Upload a data model from the Data Models management page. You can only upload one data model at a time.

Splunk software validates any file that you try to upload. It cannot upload files that contain anything other than valid JSON data model code.

Steps

  1. Navigate to the Data Models management page.
  2. Click Upload Data Model.
  3. Identify the JSON File that you want to upload.
    The ID field populates with the original ID of the data model.
  4. (Optional) Change the data model ID to a new, unique value.
    Keep in mind that once you save the data model file to your system you will not be able to change this ID. You can still edit the data model title after you save it to your system.
  5. Provide the name of the App that the data model belongs to.
  6. (Optional) If your capabilities allow it, change the uploaded data model permissions from Private to Shared in App.
    • Shared in App indicates that the data model is shared with all users of the App.
    • If you select Shared in App you can also enable acceleration for the data model by selecting Accelerate and choosing a Summary Range.
  7. Click Upload to upload the data model.
    The uploaded data model appears in the Data Model management page listing if it passes validation.

See About data model permissions.

See Enable data model acceleration.

Delete a data model

You can delete a data model from the Data Model Editor or the Data Models management page.

If your role grants you the ability to create data models, it should grant you the ability to delete them as well. For more information, see Enable roles to create data models.

Delete a data model from the Data Model Editor

  1. In the Search & Reporting app, click Datasets to open the Datasets listing page..
  2. Locate a data model dataset that belongs to the dataset that you want to delete.
  3. Select Manage > Edit Dataset.
  4. In the Data Model Editor, select Edit > Delete.

Delete a data model from the Data Models management page

  1. In the Search & Reporting app, click Datasets to open the Datasets listing page..
  2. Locate a data model dataset that belongs to the dataset that you want to delete.
  3. Select Manage > Edit Dataset.
  4. Click All Data Models.
  5. Locate the data model that you want to delete.
  6. Select Manage > Edit.

Manual data model management

Splunk does not recommend that you manage data models manually by hand-moving their files or hand-coding data model files. You should create and edit data models in Splunk Web. When you edit models in Splunk Web the Data Model Editor validates your changes. The Data Model Editor cannot validate changes in models created or edited manually.

Data models are stored on disk as JSON files. They have associated configs in datamodels.conf and metadata in local.meta (for data models that you create) and default.meta (for data models delivered with the product).

Data models that you create are stored in <yourapp>/local/data/models, while data models delivered with the product can be found in <yourapp>/default/data/models.

You can manually move model files between Splunk implementations but it's far easier to use the Data Model Download and Upload feature in Splunk Web. If you absolutely must move model files manually, take care to move their datamodels.conf stanzas and local.meta metadata when you do so.

The same goes for deleting data models. In general it's best to do it through Splunk Web so the appropriate cleanup is carried out.

PREVIOUS
About data models
  NEXT
Design data models

This documentation applies to the following versions of Splunk® Enterprise: 6.5.0, 6.5.1, 6.5.1612 (Splunk Cloud only), 6.5.2, 6.5.3, 6.5.4, 6.5.5, 6.6.0, 6.6.1, 6.6.2


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters