Review your apps and add-ons

If you inherited a Splunk Enterprise deployment for a large organization, you might have many apps and add-ons running on your system. This topic provides an overview of Splunk apps and add-ons, and helps you to identify the apps and add-ons installed on your Splunk Enterprise instance.

It is important to identify any Splunk Premium Solution apps running on your system. These apps provide comprehensive data analysis for specific use cases, such as IT operations and security, and can require additional resources and management.

Splunk app and add-on overview

Splunk apps and add-ons are packaged sets of configuration files that you install on a Splunk Enterprise instance. Apps and add-ons are defined as follows:

Apps: Splunk apps provide user interfaces that let you work with your data. Apps often use one or more add-ons to ingest different types of data. See Apps and add-ons in the Splunk Enterprise Admin Manual.

Add-ons: Add-ons enable Splunk Enterprise, or a Splunk app, to ingest or map a particular type of data. See About Splunk add-ons in the Splunk Add-ons manual.

All Splunk apps and add-ons run on Splunk Enterprise. Splunk Enterprise includes the Splunk Search and Reporting app. This app provides the core search environment for Splunk Enterprise, and lets you create and manage Splunk knowledge objects, such as saved searches, reports, alerts, dashboards, datasets, and so on.

Deployment requirements and considerations

Splunk apps and add-ons run on any supported Splunk Enterprise deployment topology, including single-instance, distributed, clustered, and cloud environments. To learn more about your existing Splunk Enterprise deployment topology, see Deployment topologies in this manual.

To familiarize yourself with any special requirements and considerations for your apps and add-ons, review the documentation for the specific app or add-on. To access documentation for all supported Splunk apps and add-ons, see Splunk Documentation.

For more information, see:

• App deployment overview in the Admin Manual.
• Where to install Splunk Add-ons in the Splunk Add-ons manual.

Survey your apps and add-ons

You can view all apps and add-ons installed on your system by using Splunk Web, which is the Splunk Enterprise UI, or by using the command line to navigate the file system on the search head.

View your apps and add-ons in Splunk Web

The Manage apps page in Splunk Web gives you access to all apps and add-ons installed in your deployment. You can view information about the app, including app name, folder name, and version. You can also enable or disable the app, set app permissions using role-based access controls, and perform actions such as edit properties and view objects.

1. Open Splunk Web.
   All enabled apps appear in the App column at left.
2. Click Apps > Manage Apps.
   The Manage Apps page opens.
3. Review the list of apps and add-ons installed on your Splunk Enterprise instance.

For more information, see:

• View app and add-on objects in this topic.
• Edit app and add-on properties in the Admin Manual.

View your apps and add-ons using the file system on the Search Head

1. Log in to a search head.
2. Navigate to the directory $SPLUNK_HOME/etc/apps.
   All of the apps and add-ons that are installed on your system are located in the apps directory.
3. Review your apps and add-ons.

App and add-on naming conventions

Splunk app folder names use a variation or abbreviation of the app product name. The following table provides some examples.

App name	App folder name
Splunk Enterprise Security	SplunkEnterpriseSecuritySuite
Splunk IT Service Intelligence	itsi

Splunk add-on folder names generally use one of the following prefixes:

Add-on prefix	Add-on description	Example name
TA	Splunk technology add-on	Splunk_TA_stream
SA	Splunk supporting add-on	SA-ITOA
DA	Splunk domain add-on (ITSI module)	DA-ITSI-OS

View app and add-on objects

When you create an app or add-on, Splunk Enterprise creates a collection of objects that makes up the app or add-on. These objects can include views, commands, navigation items, event types, saved searches, reports, and so on.

In addition, each app object has role-based permissions associated with it that determine who can view or edit the object. By default, the Splunk Enterprise admin user has write permissions and can edit all objects across the system.

Use Splunk Web to view all objects that pertain to a specific app or add-on, as follows:

1. In Splunk Web, click Settings > All configurations.
2. In the App context menu, select the name of the app whose objects you want to view.
3. Select the Show only objects created in this app context check box.

For more information, see:

• Manage app and add-on objects in the Admin Manual.
• App architecture and object ownership in the Admin Manual.

Identify apps that use the KV store

The KV store resides on every Splunk Enterprise version 6.2 or later instance by default and is often active on search heads. KV store can maintain state information about apps. In addition, some apps, like Enterprise Security, use the KV store for lookups. KV store replicates its data across search heads using port 8191 by default. KV store processes are independent of a search head cluster's processes.

Discover KV store members using the Splunk command line interface. See About the CLI in the Admin Manual.

1. Log in to a search head.
2. Type ./splunk show kvstore-status

Make note of the following:

• Whether disabled is 1 or 0.
• Which nodes are members of the KV store cluster.
• The port number that KV store is using.

Add the KV store members and port numbers to your deployment diagram. This command also returns information on which node is captain, but this information is not useful at this stage. Captaincy can change, so leave this detail off of your diagram.

Next, determine which apps, if any, use the KV store.

Apps that use the KV store have collections.conf defined in $SPLUNK_HOME/etc/apps/<app name>/default. In addition, transforms.conf has references to the collections with external_type = kvstore.

For a list of apps that have collections defined:

1. Log in to a search head.
2. At the command line, from the Splunk installation directory, type ./splunk btool collections list --debug
3. In the results, look for items in $SPLUNK_HOME/etc/apps

For more information, see:

• Manage state with the key value store on the developer portal.
• Configure KV store lookups in the Knowledge Manager Manual.
• About the app key value store in the Admin Manual.
• KV store troubleshooting tools in the Admin Manual.

Identify deployment apps

Distributed Splunk Enterprise deployments use the deployment server to distribute app and configuration file updates to groups of Splunk Enterprise components, such as forwarders, non-clustered indexers, and search heads. These apps and configuration files are called deployment apps. Deployment apps reside on a Splunk Enterprise instance that has been assigned the deployment server role, and are located in the directory $SPLUNK_HOME/etc/deployment-apps.

View your deployment apps in Splunk Web:

1. Identify which Splunk Enterprise instance is assigned the deployment server role. For help discovering the correct Splunk Enterprise instance, see Discover management components in this manual.
2. Log in to the deployment server.
3. Click Settings > Forwarder management.
4. On the Forwarder Management page, note the following:
   • Apps. Apps are the deployment apps currently being distributed by the deployment server.
   • Clients. Clients are the remote Splunk Enterprise instances to which the deployment server distributes the deployment apps.
   • Server Classes. Server classes are groups of deployment clients. The server class determines the specific set of clients that receive the app update.
5. Record server classes on your deployment diagram.

View your deployment apps using the file system on the deployment server:

1. Log in to the machine hosting the deployment server.
2. Go to $SPLUNK_HOME/etc/deployment-apps
3. Make note of the apps currently being distributed by the deployment server.

For more information, see:

• About deployment server and forwarder managment in Updating Splunk Enterprise Instances.
• Deploy Apps to client in Updating Splunk Enterprise Instances.

For information on deploying apps to search head clusters, indexers, and indexer clusters, see:

• Use the deployer to distribute apps and configuration updates in Distributed Search.
• Update common peer configurations and apps in Managing Indexers and Clusters of Indexers.

Download apps from Splunkbase

Splunk offers a large number of apps and add-ons, free and for purchase, that can help you extend your data ingestion, search, and analysis capabilities. Splunk apps and add-ons are available for download at Splunkbase.

Splunk premium solutions overview

Splunk premium solutions are apps developed by Splunk that provide comprehensive data search and analysis capabilities for specific use cases, such as IT operations analytics, and security threat detection and analysis.

Splunk premium solutions include:

• Splunk Enterprise Security (ES)
• Splunk IT Service Intelligence (ITSI)
• Splunk app for PCI Compliance

For more information, see Splunk premium solutions or contact your Splunk account representative.

ES and ITSI requirements and considerations

Splunk ES and ITSI production deployments can be resource intensive. Depending on several factors, such as the number of concurrent searches, the daily index volume, and the unused capacity of your environment, additional hardware might be required above the baseline Splunk Enterprise hardware. For the latest Splunk Enterprise hardware requirements, see Reference hardware in the Splunk Enterprise Capacity planning manual.

Familiarize yourself with the factors that affect ES and ITSI performance, including the respective number of correlation or KPI searches running and the number of concurrent users on the system. This will help you to evaluate the performance of your system and determine how and when to scale your deployment.

It is important to familiarize yourself with search head and indexer considerations that might impact the configuration of your deployment.

For information on ES search head and indexer considerations, as well as performance and capacity planning, see Deployment planning in the Splunk Enterprise Security Installation and Upgrade Manual

For information on ITSI search head and indexer considerations, as well as performance and capacity planning, see Plan your ITSI deployment in the Splunk ITSI Install and Upgrade Manual.

Splunk Enterprise Security overview

Splunk Enterprise Security (ES) detects patterns in your data and evaluates events for security-relevant incidents using correlation searches. When a correlation search detects a suspicious pattern, the correlation search can create a notable event. The app provides specialized dashboards and visualizations that you can use to you identify, triage, and analyze security incidents.

See the Splunk Enterprise Security documentation.

View your ES correlation searches

View the correlation searches available in Splunk Enterprise Security and those that are enabled to better understand the use cases that Splunk Enterprise Security is being used to detect. To get a list of the correlation searches enabled in Splunk Enterprise Security, you can use a REST search to view the information in a table. See List correlation searches in Splunk Enterprise Security in Administer Splunk Enterprise Security.

Content Management data model row expansion

The Content Profile dashboard is removed in favor of the Content Management data model row expansion. Enterprise Security uses data that has been mapped to CIM-specific or ES-specific data models and accelerated to produce faster search results across a broad set of technologies. Review the data models in use in your environment and get an overview of the knowledge objects that correspond to the data models. See Expand Content Management searches to view dependency and usage information in Splunk Enterprise Security.

The Data Model Audit dashboard

In addition, you can review the status of data models on the Data Model Audit dashboard and the retention and acceleration settings for data models. Data models that are not fully accelerated can result in missing or out-of-date information on dashboards or notable events in Splunk Enterprise Security. See Data Model Audit in Use Splunk Enterprise Security and Configure data models for Splunk Enterprise Security in the Splunk Enterprise Security Installation and Upgrade Manual.

Learn more about Splunk Enterprise Security

To learn more about important Splunk Enterprise Security concepts and features, see:

• Incident review in Administer Splunk Enterprise Security.
• Correlation search overview in Administer Splunk Enterprise Security.
• Add asset and identity information to Splunk Enterprise Security in Administer Splunk Enterprise Security.
• Add threat intelligence to Splunk Enterprise Security in Administer Splunk Enterprise Security.
• Risk Analysis in Use Splunk Enterprise Security.
• Accelerate your investigations with security intelligence in Use Splunk Enterprise Security.
• Monitor security domain activity in Use Splunk Enterprise Security.

Splunk IT Service Intelligence overview

Splunk IT Service Intelligence (ITSI) monitors the health of IT services using key performance indicators (KPIs) that track the severity-level of IT performance metrics. When KPI values meet threshold conditions, ITSI generates a notable event. The app provides features for aggregating and analyzing notable events, as well as dashboards and visualizations that let you continuously monitor IT services and perform root cause investigations.

See the Splunk IT Service Intelligence documentation.

View your ITSI services and KPIs

Review your services and the KPIs contained within them to understand the IT operations and business processes that your services are monitoring. KPIs help you identify the performance metrics being used to evaluate service health. KPI search properties include source search types (data model, ad hoc, or base search), calculations (search frequency and calculated stat), and severity-level thresholds that determine the KPI health status.

1. From the ITSI main menu, click Configure > Services.
2. Review the list of services.
3. Click on any service and review the list of KPIs within the service. Each KPI represents an IT performance metric, such as CPU Utilization %, Memory Free %, Response Time, and so on.
4. Select any KPI in the list and expand the Search and Calculate panel.
5. For Source, note the Threshold field. This is the field in your data for which the KPI search returns a value. For example, cpu_load_percent. Click Edit to examine the source search details. Note that base searches, such as those provided by ITSI modules, tend to provide best search performance.
6. For Entities, note the entity filtering fields. These fields determine the entities against which a KPI search runs.
7. For Calculation, note the statistic that the KPI calculates. For example, Average. Also note the KPI frequency and time range. KPIs can run every 1, 5, or 15 minutes.
8. Expand the Thresholding panel.
9. In the threshold preview graph, note the severity-level thresholds set for the KPI. When KPI values meet threshold conditions, the KPI status changes, for example, from high to critical.

For more information, see Overview of creating services in ITSI in the ITSI Service Insights manual.

Review associated entities

Identify the entities associated with your services. Entities are IT components that act as the primary data sources for ITSI services. KPI searches run against entities based on filtering conditions that you define. In more complex ITSI deployments a single entity can be associated with multiple services and have multiple different KPIs running against it.

To view entities associated with a service:

1. From the ITSI main menu, click Configuration > Entities.
2. Review the list of entities. In the services column, note the services associated with each entity.
3. For any entity in the list, click View Health.
4. Review the entity analysis dashboards as well as the services the entity is associated with, the KPIs running against the entity, and the associated notable events.

For more information, see Overview of entity integrations in ITSI in the ITSI Entity Integrations manual.

View all ITSI KPIs

Use Splunk Web to view all KPI searches running on the search head. This will give you an idea of the number of concurrent searches contributing to the search load. You can view additional information, including the KPI search string, search frequency, time range, and run times for recent KPI search jobs.

1. In Splunk Web, click Setttings > Search, reports, and alerts.
2. Select the Show only objects created in this app context checkbox.
   All apps created in the ITSI app context appear in the list. KPI search names use the following syntax:

   Indicator - <KPI_id> - ITSI Search
   

   For example

   Indicator - 3bee62acf7f4de2a095e475f - ITSI Search
   

3. For any KPI search, click View Recent. Note the KPI run time.
4. Click on the name of the KPI search. Note the KPI search string, time range, and schedule.

Be aware that average KPI run time, KPI frequency, and the number of entities referenced per KPI, along with the total number of concurrent searches running on the system can markedly impact performance. For more information, see Performance considerations in the ITSI Install and Upgrade manual.

About Splunk User Behavior Analytics

Splunk User Behavior Analytics (UBA) helps you find known, unknown, and hidden threats in your environment. You can use Splunk UBA to visualize and investigate internal and external threats and anomalies. Splunk UBA integrates with Splunk Enterprise Security to take advantage of Splunk events and to investigate UBA threats alongside other notable events in your organization.

See the Splunk User Behavior Analytics documentation.