Splunk® Enterprise

Knowledge Manager Manual

Download manual as PDF

This documentation does not apply to the most recent version of Splunk. Click here for the latest version.
Download topic as PDF

Design data models and objects

In Splunk Web, you use the Data Model Editor to design new data models and edit existing models. This topic shows you how to use the Data Model Editor to create and edit:

  • Build out data model object hierarchies by adding root objects and child objects to data models.
  • Define object datasets (by providing constraints, search strings, or transaction definitions).
  • Rename objects.
  • Delete objects.

You can also use the Data Model Editor to create and edit object attributes. For more information, see "Define object attributes."

Note: This topic will not spend much time explaining basic data model concepts. If you have not worked with data models in Splunk Enterprise before you may find it helpful to review the topic "About data models," in this manual. It provides a lot of background detail around what data models and data model objects actually are and how they work.

For information about creating and managing new data models, see "Manage data models," in this manual. Aside from creating new data models via the Data Models management page, this topic will also show you how to manage data model permissions and acceleration.

The Data Model Editor

Data models are collections of data model objects arranged in hierarchical structures. To design a new data model or redesign an existing data model, you go to the Data Model Editor. In the Data Model Editor, you can create objects for your data model, define their constraints and attributes, arrange them in logical object hierarchies, and maintain them.

Cupk data model builder.png

Manage objects in an existing data model

There are five ways to access the Data Model Editor for an existing data model:

  • In Pivot, on the Select a Data Model page, expand the row for a data model to reveal detail information for that model. Click Edit Objects to access the Data Model Editor for that data model.
  • In Pivot, select a data model. On the Select an Object page for that data model, click Edit Objects.
  • In Pivot, select a data model and then select an object from that data model. On the New Pivot page, click Data Source and select Edit Object.
  • Navigate to System > Data Models, click the Edit dropdown for the data model you want to edit, and select Edit Objects.
    • Alternatively, navigate to System > Data Models, expand the row for the data model you want to edit, and select Edit for the listed objects.

Note: You can only edit a specific data model if your permissions enable you to do so.

Add a root event object to a data model

Data models are typically composed chiefly of object hierarchies built on root event objects. Each root event object represents a set of data that is defined by a constraint: a simple search that filters out events that aren't relevant to the object. Constraints look like the first part of a search, before pipe characters and additional search commands are added.

Constraints for root event objects are usually designed to return a fairly wide range of data. A large dataset gives you something to work with when you associate child event objects with the root event object, as each child event object adds an additional constraint to the ones it inherits from its ancestor objects, narrowing down the dataset that it represents.

Note: For more information on how constraints work to narrow down datasets in an object hierarchy, see the section on object constraints in the "About data models" topic, in this manual.

To add an root event object to your data model, click Add Object in the Data Model Editor and select Root Event. This takes you to the Add Event Object page.

Bubb dm addeventobject.png

Give the root event object an Object Name, Object ID, and one or more Constraints.

The Object Name field can accept any character except asterisks. It can also accept blank spaces between characters. It's what you'll see on the Choose an Object page and other places where data model objects are listed.

The Object ID must be a unique identifier for the object. It cannot contain spaces or any characters that aren't alphanumeric, underscores, or hyphens (a-z, A-Z, 0-9, _, or -). Spaces between characters are also not allowed. Once you save the Object ID value you can't edit it.

After you provide Constraints for the root event object you can click Preview to test whether the constraints you've supplied return the kinds of events you want.

Add a root transaction object to a data model

Root transaction objects enable you to create object hierarchies that are based on a dataset made up of transaction events. A transaction event is actually a collection of conceptually-related events that spans time, such as all website access events related to a single customer hotel reservation session, or all events related to a firewall intrusion incident. When you define a root transaction object, you define the transaction that pulls out a set of transaction events.

Read up on transactions and the transaction command if you're unfamiliar with how they work. Get started at About transactions, in the Search Manual. Get detail information on the transaction command at its entry in the Search Reference.

Note: Root transaction objects and their children do not benefit from data model acceleration.

To add a root transaction object to your data model, click Add Object in the Data Model Editor and select Root Transaction. This takes you to the Add Transaction Object page.

Bubbles dm addtransobj.png

Root transaction object definitions require an Object Name and Object ID and at least one Group Object. The Group by, Max Pause, and Max Span fields are optional, but the transaction definition is incomplete until at least one of those three fields is defined.

The Object Name field can accept any character except asterisks. It can also accept blank spaces between characters. It's what you'll see on the Choose an Object page and other places where data model objects are listed.

The Object ID must be a unique identifier for the object. It cannot contain spaces or any characters that aren't alphanumeric, underscores, or hyphens (a-z, A-Z, 0-9, _, or -). Spaces between characters are also not allowed. Once you save the Object ID value you can't edit it.

All root transaction object definitions require one or more Group Object names to define the pool of data from which the transaction object will derive its transactions. There are restrictions on what you can add under Group Object, however. Group Object can contain one of the following three options:

  • One or more event objects (either root event objects or child event objects)
  • One transaction object (either root or child)
  • One search object (either root or child)

In addition, you are restricted to objects that exist within the currently selected data model.

If you're familiar with how the transaction command works, you can think of the Group Objects as the way we provide the portion of the search string that appears before the transaction command. Take the example presented in the preceding screenshot, where we've added the Apache Access Search object to the definition of the root transaction object Web Session. Apache Access Search represents a set of successful webserver access events--its two constraints are status < 600 and sourcetype = access_* OR source = *.log. So the start of the transaction search that this root transaction object represents would be:

status<600 sourcetype=access_* OR source=*.log | transaction...

Now we only have to define the rest of the transaction argument.

Group by is where you can list fields for the root transaction object. If you choose two or more fields you're saying that the transaction will find events that share the same values for the chosen fields. The Group by dropdown does not appear until you identify at least one Group Object, and it only allows you to choose attributes that belong to the group object or objects that you have listed. When you have multiple objects selected, Group by only allows you to select from the set of attributes that are shared by all of the selected objects.

Note: While the Group by fields are attributes that belong to the Group Object(s) selected for the root transaction object, selecting them does not add them to the root transaction object as attributes. They are used to define the transaction.

You can change the list order of Group by fields. In edge cases where you have events within a transaction with the same timestamp, field reordering can affect how Splunk Enterprise sorts those events.

The duration-based options, Max Pause and Max Span, have the same function as the maxpause and maxspan arguments in the transaction search command. Both of these settings help you to ensure that the transactions found by your transaction object do not contain erroneous events.

The Max Pause field enables you to set a limit to the amount of time that can separate the events that make up a single transaction. For example, you may know that a certain transaction shouldn't contain any events with timestamps that are more than 20 seconds apart--if a matching event does come in more than 20 seconds after the last one, it might signal the start of a new transaction event, or it could just be an outlying event. To ensure that your transaction object returns transactions that follow this rule, set the value of Max Pause to 20 seconds.

The Max Span field enables you to control the overall span of time that a particular transaction encompasses. For example, if you know that the transactions you're looking for never span more than 5 minutes, you could set 5 minutes as the Max Span value for the corresponding transaction object.

So to return to our example in the screenshot, if we were to represent transaction object as a search string, it would look like this:

status<600 sourcetype=access_* OR source=*.log | transaction clientip useragent maxspan=1m

This looks through the data returned by the status<600 sourcetype=access_* OR source=*.log and seeks out "web session" transactions: transactional groupings of events with identical values for clientip and useragent where the duration of the transaction as a whole does not exceed one minute.

After you define your root transaction object's Group by you can click Preview to test whether the constraints you've supplied return the kinds of events you want.

Add a root search object to a data model

Root search objects enable you to create object hierarchies where the base dataset is the result of an arbitrary transforming search--a search that uses statistical commands to define a base dataset where one or more fields aggregate over the entire dataset.

The results returned by root search objects and their children are essentially table rows (unless the search is not a transforming search, in which case events are returned, just as they are for root event objects and their children).

Note: We suggest that whenever possible, you avoid using root search objects to create an object hierarchy, because root search objects and their children do not benefit from data model acceleration. There are many kinds of searches that can be set up as root event objects. For more information, see "When not to create root search objects," below.

To add a root search object to your data model, click Add Object in the Data Model Editor and select Root Search. This takes you to the Add Search Object page.

Cupk base search obj.png

Give the root search object an Object Name, Object ID, and search string. To preview the results of the search in the section at the bottom of the page, click the magnifying glass icon to run the search, or just hit return on your keyboard while your cursor is in the search bar.

The Object Name field can accept any character except asterisks. It can also accept blank spaces between characters. It's what you'll see on the Choose an Object page and other places where data model objects are listed.

The Object ID must be a unique identifier for the object. It cannot contain spaces or any characters that aren't alphanumeric, underscores, or hyphens (a-z, A-Z, 0-9, _, or -). Spaces between characters are also not allowed. Once you save the Object ID, value you can't edit it.

For more information about designing search strings, see the Search Manual.

When not to create root search objects

As we stated above, you should avoid creating root search objects if you can design an root event object that will do the job. Root event objects have a major advantage over root search objects: Event object hierarchies can be accelerated, while search object hierarchies cannot. If you accelerate your data model and then select an object from that model to run a Pivot operation with (such as the generation of a chart visualization), the operation will complete much faster if you can use an object from an event object hierarchy than it would if you used an object from a search object hierarchy.

Don't create a root search object for your search if:

  • The search only involves the eval, rex, or lookup commands (or some combination of those three commands). You can set this sort of search up as an root event search with the help of the Eval Expression, Regular Expression, and Lookup attributes.
For example, say you have this search:

sourcetype=access_* | eval userid=clientip+useragent

You can utilize this search in a root event object by first defining its constraint as sourcetype=access_*. Save the object, and then, in the Data Model Editor for that object, add an Eval Expression attribute to the object using userid=clientip+useragent as the attribute definition.
  • You would like to get the benefits of data model acceleration but you want to use commands that can be configured in .conf files, so they run automatically as events are indexed. Many commands that extract fields can be configured in this way, meaning that the fields they extract will be available as auto-extracted attributes when you define the data model.
    • Many commands can be set up in props.conf and transforms.conf to aid in field extraction.
    • Additionally, if you typically use the multikv command to extract fields from .csv files and the like, you can configure multikv.conf to extract fields as the .csv file is indexed.
    • If you do not need to accelerate the underlying root-level object, you can create a root search object that uses these commands in its defining search.
  • You want to create line or area charts in Pivot. These chart types require that _time be an auto-extracted attribute. Because root search objects are designed to return table rows for transforming searches, they do not auto-extract _time as an attribute.
  • The search is a simple transaction search. Set it up as a root transaction object.

You should create root search objects for any searches that do not map directly to Splunk events. In other words, searches that involve input or output that is not in the format of an event. This includes searches that:

  • Make use of transforming commands such as stats, chart, and timechart. Transforming commands organize the data they return into tables rather than event lists.
  • Use other commands that do not return events.
  • Pull in data from external non-Splunk sources using a command other than lookup. This data cannot be guaranteed to have default fields like host, source, sourcetype, or _time and therefore might not be event-mappable. An example would be using the inputcsv command to get information from an external .csv file.

Add a child object to a data model

You can add child objects to root objects and other child objects. A child object inherits all of the constraints and attributes that belong to its parent object. A single object can be associated with multiple child objects

When you define a new child object, you give it one or more additional constraints, to further focus the dataset that the object represents. For example, if your Web Intelligence data model has a root event object called HTTP Request that captures all webserver access events, you could give it three child event objects: HTTP Success, HTTP Error, and HTTP Redirect. Each child event object focuses on a specific subset of the HTTP Request dataset:

  • The child event object HTTP Success uses the additional constraint status = 2* to focus on successful webserver access events.
  • HTTP Error uses the additional constraint status = 4* to focus on failed webserver access events.
  • HTTP Redirect uses the additional constraint status = 3* to focus on redirected webserver access events.

The addition of attributes beyond those that are inherited from the parent object is optional. For more information about attribute definition, see "Manage Object attributes with the Data Model Editor," below.

To add a child object to your data model, select the parent object in the left-hand object hierarchy, click Add Object in the Data Model Editor, and select Child. This takes you to the Add Child Object page.

Cupk add child obj.png

Give the child object an Object Name and Object ID.

The Object Name field can accept any character except asterisks. It can also accept blank spaces between characters. It's what you'll see on the Choose an Object page and other places where data model objects are listed.

The Object ID must be a unique identifier for the object. It cannot contain spaces or any characters that aren't alphanumeric, underscores, or hyphens (a-z, A-Z, 0-9, _, or -). Spaces between characters are also not allowed. Once you save the Object ID value you can't edit it.

After you define a Constraint for the child object you can click Preview to test whether the constraints you've supplied return the kinds of events you want.

Some best practices for data model design

It can take some trial and error to determine the data model designs that work for you. Here are some tips that can get you off to a running start.

  • Use root event objects whenever possible to take advantage of the benefits of data model acceleration (and to benefit from their ease of optimization).
  • Minimize object hierarchy depth whenever possible. Constraint-based filtering is less efficient deeper down the tree.
  • Put the root event object with the deepest tree (and most matching results) first. Data model acceleration applies only to the first root event object and its children.
  • When defining constraints for a root event object that will be accelerated, when possible include the index or indexes it is selecting from. Data model acceleration efficiency is improved when the data model isn't searching across all of your indexes (this is what it does by default when indexes are not specified).
  • Use attribute flags to selectively expose small groups of attributes for each object. You can expose and hide different attributes for different objects. A child attribute can expose an entirely different set of attributes than those exposed by its parent. Your Pivot users will benefit from this selection by not having to deal with a bewildering array of attributes whenever they set out to make a pivot chart or table. Instead they'll see only those attributes that make sense in the context of the object they've chosen.
  • Reverse-engineer your existing dashboards and searches into data models. This can be a way to quickly get started with data models. Dashboards built with pivot-derived panels are easier to maintain.
  • When designing a new data model, first try to understand what your Pivot users hope to be able to do with it. Work backwards from there. The structure of your model should be determined by your users' needs and expectations.
PREVIOUS
Manage data models
  NEXT
Define object attributes

This documentation applies to the following versions of Splunk® Enterprise: 6.1, 6.1.1, 6.1.2, 6.1.3, 6.1.4, 6.1.5, 6.1.6, 6.1.7, 6.1.8, 6.1.9, 6.1.10, 6.1.11, 6.1.12, 6.1.13, 6.1.14


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters