Splunk® Content Packs for ITSI and IT Essentials Work

Splunk Content Packs for ITSI and IT Essentials Work

Acrobat logo Download manual as PDF


Acrobat logo Download topic as PDF

Install and configure the Content Pack for Monitoring and Alerting

Perform the following high-level steps to install and configure the Content Pack for Monitoring and Alerting:

  1. Install third-party apps from Splunkbase.
  2. Install the Conf Editor app (if onboarding external alerts with Universal Alerting).
  3. Install the Add-on for Content Pack for Monitoring and Alerting.
  4. Update the itsi_kpi_attributes lookup.
  5. Install the content pack on your ITSI search head.
  6. Enable the appropriate correlation searches and aggregation policies.

Prerequisite

Create a full backup of your ITSI environment in case you need to uninstall the content pack later. See Create a full backup.

Step 1: (Optional) Install third-party apps from Splunkbase

While not required, this content pack leverages several Splunkbase apps to help you manage and visualize alerting data. It's a best practice to install each of the following apps.

  • Lookup File Editor app. The Content Pack for Monitoring and Alerting uses several new lookup files. The files enrich notable events with the information necessary to group related events, drive alert actions, and engage the correct stakeholders. The Lookup File Editor lets you create and maintain this information in your ITSI environment. After installing this app, you must immediately restart Splunk software.
  • Punchcard Visualization app. Several dashboards within the Content Pack depend on the punchcard visualization to better visualize concentrations of data over hours of the day or days of the week. If you use the dashboards within this Content Pack, this visualization is recommended.
  • Conf Editor app. This app is useful for creating normalized alert fields for alert sources when implementing Universal Alerting. The app provides a simple GUI for creating Splunk search-time knowledge object fields ('eval' and 'extract'). After installing this app, you must immediately restart Splunk software.

Step 2: Install the Add-on for the Content Pack for Monitoring and Alerting

You must perform this step before installing the content pack, otherwise the restore fails.

This content pack depends on several Splunk knowledge objects such as dashboards, reports, lookups, and macros which are required to install and use the content pack. Download the latest version of the Add-on for the Content Pack for Monitoring and Alerting to access these knowledge objects.

Download the supporting add-on from Splunkbase and install it on your search head running ITSI. You don't need to restart Splunk software unless it's specifically indicated after the installation process.

After installing the add-on, you must perform the next step to update the itsi_kpi_attributes lookup. You receive errors for missing lookups on all searches until you complete this step.

Step 3: Update the itsi_kpi_attributes lookup

This content pack lets you store metadata about your ITSI services, KPIs, and contacts to use within your alerting configuration. The metadata attributes are stored in two new lookups called itsi_kpi_attributes and itsi_episode_contact_map that are packaged within the supporting Add-on for the Content Pack for Monitoring and Alerting. You must update these lookups after you install the add-on and on an ongoing basis to ensure that each service and KPI in your environment has a corresponding record.

The add-on includes a pre-built report that you can run to ensure the lookups remain up-to-date. To run the report, perform the following steps:

  1. From the ITSI main menu, click Dashboards > Reports (or Search > Reports in versions prior to 4.5.0).
  2. Locate the report called ITSI KPI Attributes Lookup Generator.
  3. Click Open in Search. The report runs and updates the itsi_kpi_attributes lookup with the latest services and KPIs.
  4. Go back to Reports and locate the report called ITSI Episode Contact Map Generator.
  5. Click Open in Search. The report runs and updates the itsi_episode_contact_map lookup with the latest services and KPIs.

To ensure new services and KPIs added to the environment are included within the lookup, it's a best practice to schedule the report to run automatically. To schedule the report, perform the following steps:

  1. From the ITSI main menu, click Dashboards > Reports (or Search > Reports in versions prior to 4.5.0).
  2. Locate the report called ITSI KPI Attributes Lookup Generator.
  3. Click Edit > Edit Schedule.
  4. Enable Schedule Report and configure the schedule. It's best to run the report at least once a day.
  5. Perform steps 1-4 for the ITSI Episode Contact Map Generator report.

Alternatively, you can run the reports manually each time you add a new service or KPI to your environment.

Step 4: Install the content pack

If you're a Splunk Cloud Platform customer on ITSI version 4.9.0 or higher, you can install the content pack directly through the Splunk App for Content Packs. You can also install content packs through the ITSI REST API. If you're an on-premises customer on a version lower than 4.8.0, see Install the content pack in an on-premises instance.

Install the content pack on Splunk Cloud Platform

Perform the following steps to install the Content Pack for Monitoring and Alerting from the Data Integrations page on version 4.8.x:

  1. From the ITSI main menu, click Configuration > Data Integrations.
  2. Click Add structure to your data.
  3. Select the Monitoring and Alerting content pack.
  4. Review what's included in the content pack and then click Proceed.
  5. Configure the following settings:
    Setting Description
    Choose which objects to install For a first-time installation, select the items you want to install and deselect any you're not interested in.


    For an upgrade, the installer identifies which objects from the content pack are new and which ones already exist in your environment from a previous installation. You can selectively choose which objects to install from the new version, or install them all.

    Choose a conflict resolution rule for the objects you install For upgrades or subsequent installs, decide what happens to duplicate objects introduced from the content pack. Choose from the following options:
    • Install as new - Objects are installed and any existing identical objects in your environment remain intact.
    • Replace existing - Existing identical objects are replaced with those from the new installation. Any changes you previously made to these objects are overwritten.
    Import as enabled Select whether to install objects as enabled or to leave them in their original state. It's recommended that you import objects as disabled to ensure your environment doesn't break from the addition of new content.


    This setting only applies to services, correlation searches, and aggregation policies. All other objects such as KPI base searches and saved searches are installed in their original state regardless of which option you choose.

    Add a prefix to your new objects Optionally, append a custom prefix to each object installed from the content pack. For example, you might prefix your objects with CP- to indicate they came from a content pack. This option can help you locate and manage the objects post-install.
    Backfill service KPIs Optionally backfill your ITSI environment with the previous seven days of KPI data. Consider enabling backfill if you want to configure adaptive thresholding and Predictive Analytics for the new services. This setting only applies to KPIs and not service health scores.
  6. When you're satisfied with your selections, click Install selected.
  7. Click Install to confirm the installation. When the installation completes you can view all objects that were successfully installed in your environment. A green check mark on the Data Integrations page indicates which content packs you've already installed.

Install the content pack through the REST API

On ITSI version 4.8.x you can use the itoa_interface/content_pack endpoint to install content packs through the ITSI REST API. The endpoint includes GET operations to fetch versioning information and preview the contents of the content pack, and a POST operation to install content packs.

Install the content pack on an on-premises instance

If you're on a pre-4.8.0 version of ITSI, perform the following steps to install the content pack:

  1. Download the following ITSI backup file: BACKUP-CP-MA-2.0.0.zip.
  2. On your ITSI search head, create a restore job and upload the backup file. Give the job the same name as the backup file you downloaded (for example, BACKUP-CP-MA-2.0.0. For instructions, see Restore from a backup zip file.
  3. After the restore job completes, check the lister pages for correlation searches and aggregation policies to confirm that the new objects have been successfully restored to your environment. For a full list of the objects shipped in this content pack, see the release notes.

Step 5: Set up your ITSI environment

Perform the following steps after installing the Content Pack for Monitoring and Alerting.

Review and enable the aggregation policies

When setting up your environment, it's best to enable the Episodes by Alarm, Episodes by ITSI Service, and Episodes by Src policies first. Other aggregation policies support more advanced groupings.

  1. From the ITSI top menu bar, click Configuration > Notable Event Aggregation Policies.
  2. Filter the policy list to the Episodes by policies.
  3. Enable the policies that are appropriate for the monitoring you want to conduct.

For more information about these aggregation policies, and instructions on when to enable them, see About the aggregation policies in the Content Pack for Monitoring and Alerting.

Enable correlation searches if you plan to onboard external alerts as Universal Alerts

If you are planning to use Universal Alerting to onboard external alert sources (such as Nagios, Solarwinds, or Splunk Infrastructure Monitoring), enable the relevant correlation searches.

  1. From the ITSI to menu bar, click Configuration > Correlation Searches.
  2. Enable these searches:
    • Universal Correlation Search
    • Episode Monitoring - Set Episode to Highest Alarm Severity

Review and enable service monitoring correlation searches

Each Service Monitoring correlation search monitors the health of the services and KPIs within your ITSI environment. The searches create notable events based on various issues with your services, KPIs, and entities. Enable a small number of correlation searches that are appropriate for the monitoring you want to conduct across your environment. It's best to enable the Sustained Service Degradation and Sustained KPI degradation correlation searches first.

For more information about these correlation searches, and instructions on when to enable them, see About the correlation searches in the Content Pack for Monitoring and Alerting.

To enable these correlation searches, perform the following steps:

  1. From the ITSI top menu bar, click Configuration > Correlation Searches.
  2. Filter the search list to the Service Monitoring searches.
  3. Enable the searches that are appropriate for the monitoring you want to conduct.

(Optional) Enable the sample services

This content pack includes several example services to demonstrate its monitoring and alerting behavior. If you have no other services in your environment or you want to scope down the monitoring and alerting to a set of test services, enable and use the sample services.

To enable the sample services, perform the following steps:

  1. From the ITSI top menu bar, click Configuration > Services.
  2. Filter the service list to the ITSI Monitoring services.
  3. Change the status to Enabled for each of the ITSI Monitoring services.

Based on the KPIs and thresholds of these example service, expect to see the services degrade from time to time. When a service degrades, the enabled correlation searches create notable events and the aggregation policies group them into episodes in Episode Review.

Review and enable the episode monitoring correlation searches

Each Episode Monitoring correlation search monitors the episodes created in your environment. When an episode meets the conditions of the search, the correlation search creates a notable event in Episode Review and the alert actions in the aggregation policy run. It's best to enable the Episode Monitoring - Critical Notable Event added to Episode and Episode Monitoring - Set Episode to Highest Alarm Severity correlation searches first.

For more information about these correlation searches, and instructions on when to enable them, see About the correlation searches in the Content Pack for Monitoring and Alerting.

To enable these correlation searches, perform the following steps:

  1. From the ITSI top menu bar, click Configuration > Correlation Searches.
  2. Filter the search list to the Episode Monitoring searches.
  3. Review and enable the correlation searches that are appropriate for the monitoring you want to conduct across your environment.

By default, the aggregation policy alert actions provided with this content pack are only configured to add a comment to the episode, except Episode Monitoring - Set Episode to Highest Alarm Severity. Review and modify the alert actions in the Action Rules section of the notable event aggregation policies you enable to take more meaningful actions. For more information, see Configure alerts in the Content Pack for Monitoring and Alerting.

Next steps

After you enable the Universal Correlation Search and recommended Notable Event Aggregation Policies, external alerts which have been normalized as Universal Alerts will be "found" and onboarded as Notable Events, then grouped as Episodes. For more details about how to normalize external alert sources, see About Universal Alerting.

After you enable one or more episode monitoring correlation searches, ITSI begins to continuously monitor newly created episodes. When an episode meets the alert criteria for that correlation search, the search generates a notable event in Episode Review and the corresponding action executes in accordance with the action rule in the aggregation policy.

Next, configure the aggregation policy to proactively send a notification to the accountable group, such as an email or a ticket in an external ticketing system. For instructions, see Configure alerts in the Content Pack for Monitoring and Alerting.

Last modified on 14 October, 2021
PREVIOUS
Release notes for the Content Pack for Monitoring and Alerting
  NEXT
Upgrade from a previous version of the Content Pack for Monitoring and Alerting

This documentation applies to the following versions of Splunk® Content Packs for ITSI and IT Essentials Work: current


Was this documentation topic helpful?

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters