Install and configure the Content Pack for ITSI Monitoring and Alerting

Perform the following high-level steps to install and configure the Content Pack for ITSI Monitoring and Alerting:

1. (Optional) Install third-party apps from Splunkbase.
2. Install the Add-on for Content Pack for ITSI Monitoring and Alerting.
3. Update the itsi_kpi_attributes lookup.
4. Install the content pack on your ITSI search head.
5. Enable the appropriate correlation searches and aggregation policies.

Prerequisite

Create a full backup of your ITSI environment in case you need to uninstall the content pack later. See Create a full backup.

(Optional) Install third-party apps from Splunkbase

While not required, this content pack leverages several Splunkbase apps to help you manage and visualize alerting data. It's a best practice to install each of the following apps.

• Lookup File Editor app. The Content Pack for ITSI Monitoring and Alerting uses several new lookup files. The files enrich notable events with the information necessary to group related events, drive alert actions, and engage the correct stakeholders. The Lookup File Editor lets you create and maintain this information in your ITSI environment. After installing this app, you must immediately restart Splunk software.
• Punchcard Visualization app. Several dashboards within the content pack depend on the punchcard visualization to better visualize concentrations of data over hours of the day or days of the week. If you plan to use the dashboards within this content pack, install this visualization.

Install the Add-on for the Content Pack for ITSI Monitoring and Alerting

You have to install the add-on as it contains the knowledge objects used by the content pack. The content pack installation might fail if the add-on isn't installed.

This content pack depends on several Splunk knowledge objects such as dashboards, reports, lookups, and macros which are required to install and use the content pack. Download the latest version of the Add-on for the Content Pack for ITSI Monitoring and Alerting to access these knowledge objects.

Download the supporting add-on from Splunkbase and install it on your search head running ITSI. You don't need to restart Splunk software unless it's specifically indicated after the installation process.

After installing the add-on, you must perform the next step to update the itsi_kpi_attributes lookup. You receive errors for missing lookups on all searches until you complete this step.

Update the itsi_kpi_attributes lookup

This content pack lets you store metadata about your ITSI services, KPIs, and contacts to use within your alerting configuration. The metadata attributes are stored in two new lookups called itsi_kpi_attributes and itsi_episode_contact_map that are packaged within the supporting Add-on for the Content Pack for ITSI Monitoring and Alerting. You must update these lookups after you install the add-on and on an ongoing basis to ensure that each service and KPI in your environment has a corresponding record.

The add-on includes a pre-built report that you can run to ensure the lookups remain up-to-date. To run the report, perform the following steps:

1. From the ITSI main menu, click Dashboards > Reports (or Search > Reports in versions prior to 4.5.0).
2. Locate the report called ITSI KPI Attributes Lookup Generator.
3. Click Open in Search. The report runs and updates the itsi_kpi_attributes lookup with the latest services and KPIs.
4. Go back to Reports and locate the report called ITSI Episode Contact Map Generator.
5. Click Open in Search. The report runs and updates the itsi_episode_contact_map lookup with the latest services and KPIs.

To ensure new services and KPIs added to the environment are included within the lookup, it's a best practice to schedule the report to run automatically. To schedule the report, perform the following steps:

1. From the ITSI main menu, click Dashboards > Reports (or Search > Reports in versions prior to 4.5.0).
2. Locate the report called ITSI KPI Attributes Lookup Generator.
3. Click Edit > Edit Schedule.
4. Enable Schedule Report and configure the schedule. It's best to run the report at least once a day.
5. Perform steps 1-4 for the ITSI Episode Contact Map Generator report.

Alternatively, you can run the reports manually each time you add a new service or KPI to your environment.

Install the content pack

The Content Pack for ITSI Monitoring and Alerting is automatically available for installation once you have installed the Splunk App for Content Packs on the search head with ITSI 4.9.0 or higher. If you are using ITSI version 4.8.x or previous, you need to install the content pack using backup and restore functionality provided by ITSI, see Install the content pack in ITSI v4.8 or previous.

Install the content pack on ITSI v4.9.0 or higher

Follow these steps to install the Content Pack for ITSI Monitoring and Alerting from the Data Integrations page on ITSI v4.9.0 or higher:

1. From the ITSI main menu, click Configuration > Data Integrations.
2. Click Add structure to your data.
3. Select the ITSI Monitoring and Alerting content pack.
4. Review what's included in the content pack and then click Proceed.
5. Configure the settings:
   • Choose which objects to install: For a first-time installation, select the items you want to install and deselect any you're not interested in. For an upgrade, the installer identifies which objects from the content pack are new and which ones already exist in your environment from a previous installation. You can selectively choose which objects to install from the new version or install all objects.
   • Choose a conflict resolution rule for the objects you install: For upgrades or subsequent installs, decide what happens to duplicate objects introduced from the content pack. Choose from these options:
     • Install as new: Any existing identical objects in your environment remain intact.
     • Replace existing: Existing identical objects are replaced with those from the new installation. Any changes you previously made to these objects are overwritten.
   • Import as enabled: Select whether to install objects as enabled or leave them in their original state. We recommend that you import objects as disabled to ensure your environment doesn't break from the addition of new content. This setting only applies to services, correlation searches, and aggregation policies. All other objects such as KPI base searches and saved searches are installed in their original state regardless of the option you choose.
   • Add a prefix to your new objects: Optionally, append a custom prefix to each object installed from the content pack. For example, you might prefix your objects with CP- to indicate they came from a content pack. This option can help you locate and manage the objects after installation.
   • Backfill service KPIs: Optionally backfill your ITSI environment with the previous seven days of KPI data. Consider enabling backfill if you want to configure adaptive thresholding and predictive analytics for the new services. This setting only applies to KPIs, not service health scores.
6. When you've made your selections, click Install selected.
7. Click Install to confirm the installation. When the installation completes you can view all objects that were installed in your environment. A green checkmark on the Data Integrations page indicates which content packs you've already installed.

Install the content pack on ITSI v4.8.0 or lower

If you're on ITSI v4.8.0 or lower, follow these steps to install the content pack:

1. Download the following ITSI backup file: BACKUP-CP-MA-2.0.0.zip.
2. On your ITSI search head, create a restore job and upload the backup file. Give the job the same name as the backup file you downloaded (for example, BACKUP-CP-MA-2.0.0. For instructions, see Restore from a backup zip file.
3. After the restore job completes, check the lister pages for correlation searches and aggregation policies to confirm that the new objects have been successfully restored to your environment. For a full list of the objects shipped in this content pack, see the release notes.

Install the content pack through the REST API

On ITSI version 4.8.x you can use the itoa_interface/content_pack endpoint to install content packs through the ITSI REST API. The endpoint includes GET operations to fetch versioning information and preview the contents of the content pack, and a POST operation to install content packs.

Set up your ITSI environment

Perform the following steps after installing the Content Pack for ITSI Monitoring and Alerting.

Review and enable the aggregation policies

When setting up your environment, it's best to enable the Episodes by Alarm, Episodes by ITSI Service, and Episodes by Src policies first. Other aggregation policies support more advanced groupings.

1. From the ITSI top menu bar, click Configuration > Notable Event Aggregation Policies.
2. Filter the policy list to the Episodes by policies.
3. Enable the policies that are appropriate for the monitoring you want to conduct.

For more information about these aggregation policies, and instructions on when to enable them, see About the aggregation policies in the Content Pack for ITSI Monitoring and Alerting.

Enable correlation searches if you plan to onboard external alerts as Universal Alerts

If you are planning to use Universal Alerting to onboard external alert sources (such as Nagios, Solarwinds, or Splunk Infrastructure Monitoring), enable the relevant correlation searches.

1. From the ITSI to menu bar, click Configuration > Correlation Searches.
2. Enable these searches:
   • Universal Correlation Search
   • Episode Monitoring - Set Episode to Highest Alarm Severity

Review and enable service monitoring correlation searches

Each Service Monitoring correlation search monitors the health of the services and KPIs within your ITSI environment. The searches create notable events based on various issues with your services, KPIs, and entities. Enable a small number of correlation searches that are appropriate for the monitoring you want to conduct across your environment. It's best to enable the Sustained Service Degradation and Sustained KPI degradation correlation searches first.

For more information about these correlation searches, and instructions on when to enable them, see About the correlation searches in the Content Pack for ITSI Monitoring and Alerting.

To enable these correlation searches, perform the following steps:

1. From the ITSI top menu bar, click Configuration > Correlation Searches.
2. Filter the search list to the Service Monitoring searches.
3. Enable the searches that are appropriate for the monitoring you want to conduct.

(Optional) Enable the sample services

This content pack includes several example services to demonstrate its monitoring and alerting behavior. If you have no other services in your environment or you want to scope down the monitoring and alerting to a set of test services, enable and use the sample services.

To enable the sample services, perform the following steps:

1. From the ITSI top menu bar, click Configuration > Services.
2. Filter the service list to the ITSI Monitoring services.
3. Change the status to Enabled for each of the ITSI Monitoring services.

Based on the KPIs and thresholds of these example service, expect to see the services degrade from time to time. When a service degrades, the enabled correlation searches create notable events and the aggregation policies group them into episodes in Episode Review.

Review and enable the episode monitoring correlation searches

Each Episode Monitoring correlation search monitors the episodes created in your environment. When an episode meets the conditions of the search, the correlation search creates a notable event in Episode Review and the alert actions in the aggregation policy run. It's best to enable the Episode Monitoring - Critical Notable Event added to Episode and Episode Monitoring - Set Episode to Highest Alarm Severity correlation searches first.

If you use the Episode Monitoring - Set Episode to Highest Alarm Severity correlation search and you want to have episodes automatically closed, you have to enable the Service_Monitoring_-_Degraded_Service_or_KPI_Returns_to_Normal correlation search for normal events to be added to the episode and thus close the episode.

For more information about these correlation searches, and instructions on when to enable them, see About the correlation searches in the Content Pack for ITSI Monitoring and Alerting.

To enable these correlation searches, perform the following steps:

1. From the ITSI top menu bar, click Configuration > Correlation Searches.
2. Filter the search list to the Episode Monitoring searches.
3. Review and enable the correlation searches that are appropriate for the monitoring you want to conduct across your environment.

By default, the aggregation policy alert actions provided with this content pack are only configured to add a comment to the episode, except Episode Monitoring - Set Episode to Highest Alarm Severity. Review and modify the alert actions in the Action Rules section of the notable event aggregation policies you enable to take more meaningful actions. For more information, see Configure alerts in the Content Pack for ITSI Monitoring and Alerting.

Next steps

After you enable the Universal Correlation Search and recommended Notable Event Aggregation Policies, external alerts which have been normalized as Universal Alerts will be "found" and onboarded as Notable Events, then grouped as Episodes. For more details about how to normalize external alert sources, see About Universal Alerting.

After you enable one or more episode monitoring correlation searches, ITSI begins to continuously monitor newly created episodes. When an episode meets the alert criteria for that correlation search, the search generates a notable event in Episode Review and the corresponding action executes in accordance with the action rule in the aggregation policy.

Next, configure the aggregation policy to proactively send a notification to the accountable group, such as an email or a ticket in an external ticketing system. For instructions, see Configure alerts in the Content Pack for ITSI Monitoring and Alerting.