Install and configure the Content Pack for Monitoring Unix and Linux

Perform the following high-level steps to configure the Content Pack for Monitoring Unix and Linux:

1. Install the content pack on your search head.
2. Create a search macro that includes all indexes you're using for data collection.
3. (Optional) Change the module macro definition for indexes.
4. Enable entity discovery to automatically discover entities for which relevant data has been collected.
5. Tune KPI base searches
6. Tune KPI threshold levels for your environment.

Prerequisites

• You have to collect data using the Splunk Add-on for Unix and Linux to use this content pack. See Data Requirements for the Content Pack for Monitoring Unix and Linux.
• Create a full backup of your ITSI environment in case you need to uninstall the content pack later. See Create a full backup of ITSI.

Install the content pack

The Content Pack for Monitoring Unix and Linux is automatically available for installation once you have installed the Splunk App for Content Packs on the search head with ITSI. For steps to install the Splunk App for Content Packs, see Install the Splunk App for Content Packs.

After you install the Splunk App for Content Packs, you can follow these steps to install this content pack:

1. From the ITSI main menu, click Configuration > Data Integrations.
2. Select Add content packs or Add structure to your data depending on your version of ITSI.
3. Select the content pack.
4. Review what's included in the content pack and then click Proceed.
5. Configure the following settings:

   Setting	Description
   Choose which objects to install	For a first-time installation, select the items you want to install and deselect any you're not interested in. For an upgrade, the installer identifies which objects from the content pack are new and which ones already exist in your environment from a previous installation. You can selectively choose which objects to install from the new version, or install them all.
   Choose a conflict resolution rule for the objects you install	For upgrades or subsequent installs, decide what happens to duplicate objects introduced from the content pack. Choose from the following options: Install as new - Objects are installed and any existing identical objects in your environment remain intact. Replace existing - Existing identical objects are replaced with those from the new installation. Any changes you previously made to these objects are overwritten.
   Import as enabled	Select whether to install objects as enabled or to leave them in their original state. It's recommended that you import objects as disabled to ensure your environment doesn't break from the addition of new content. This setting only applies to services, correlation searches, and aggregation policies. All other objects such as KPI base searches and saved searches are installed in their original state regardless of which option you choose.
   Add a prefix to your new objects	Optionally, append a custom prefix to each object installed from the content pack. For example, you might prefix your objects with CP- to indicate they came from a content pack. This option can help you locate and manage the objects post-install.
   Backfill service KPIs	Optionally backfill your ITSI environment with the previous seven days of KPI data. Consider enabling backfill if you want to configure adaptive thresholding and Predictive Analytics for the new services. This setting only applies to KPIs and not service health scores.

6. When you're satisfied with your selections, click Install selected.
7. Click Install to confirm the installation. When the installation completes you can view all objects that were successfully installed in your environment. A green check mark in the main Content Library list indicates which content packs you've already installed.

Create the index search macro

If you're not collecting data in the default indexes given by the Splunk Add-on for Unix and Linux, you need to create a new macro with the indexes that you're using for data collection.

Prerequisites

• You have to have an admin role to create the index search macro.
• You have to know the indexes that your organization uses to send data to your Splunk deployment using the Splunk Add-on for Unix and Linux.

Steps

1. From Splunk Web, click Settings > Advanced Search > Search macros.
2. Click New Search Macro.
3. Configure the following fields:

   Field	Value
   Destination app	itsi
   Name	itsi-cp-nix-indexes
   Definition	Add all of the indexes that you're using for data collection from add-ons combined with OR operators. For example: (index=os OR index=my_nix_index OR index=<index-name>)

4. Click Save.
5. Configure read/write permissions for the macro:
1. For the newly created macro, click Permissions.
2. Select All apps (system).
3. Give Read access to Everyone.
4. Give Write access to admin.
5. Click Save.

(Optional) Change the module macro definition for indexes

The ITSI Operating System Module includes dashboards displaying OS metrics and other data. You can edit the module's search macro to populate the module's dashboards with data collected using the approaches in this content pack. Add the default indexes that you're using for data collection. For more information, see About the Operating System Module in the ITSI Modules Manual.

1. From Splunk Web, click Settings > Advanced Search > Search macros.
2. In the filter bar, search for itsi_os_module_indexes.
3. Select the itsi_os_module_indexes macro.
4. In the Definition field, add all of the indexes that you're using for data collection from add-ons combined with OR operators.
   For example:

   (index=windows OR index=perfmon OR index=os OR index=<index-name>)
   

Enable automatic entity discovery for events or metrics

Enable automatic entity discovery for events

Perform the following steps to ensure that ITSI automatically detects your Unix and Linux hosts. For best results, perform these steps after you configure at least one host to send data to Splunk Enterprise using the Splunk Add-on for Unix and Linux.

1. Navigate to ITSI on the search head.
2. Click Configuration > Entities.
3. Click Create Entity > Import from Search.
4. Select Ad hoc search and enter the following search:

   `itsi-cp-nix-indexes` (sourcetype="Unix:Version" OR source=hardware) earliest=-24h | eval role="operating_system_host" | stats latest(family) as family, latest(version) as version, latest(vendor_product) as vendor_product, latest(role) as itsi_role, latest(cpu_cores) as cpu_cores, latest(mem) as memory, latest(cpu_architecture) as cpu_architecture by host | fields + host, family, version, vendor_product, itsi_role, cpu_cores, memory, cpu_architecture

5. Click the search icon to run the search and confirm that one or more hosts are shown with all columns populated.
6. Click Next.
7. In the Import Column As column, set the host field to Entity Title. Set all other fields to Entity Information Field.
8. Set Conflict Resolution to Update Existing Entities and set the Conflict Resolution Field to host.
9. Click Import.
10. After the import completes, click Set up Recurring Import.
11. Name the recurring import ITSI discovery of Unix and Linux servers and set the frequency based on the needs of your deployment. Use Run on cron Schedule for maximum flexibility.
12. Click Submit.
    ITSI creates the new modular input in $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf.

Enable automatic entity discovery for metrics

Perform the following steps to ensure that ITSI automatically detects your Unix and Linux hosts. For best results, perform these steps after you configure at least one host to send data to Splunk Enterprise using the Splunk Add-on for Unix and Linux.

1. Navigate to ITSI on the search head.
2. Click Configuration > Entities.
3. Click Create Entity > Import from Search.
4. Select Ad hoc search and enter the following search:

   | inputlookup itsi_entities where entity_type_ids=ta_nix | rename title AS host , entity_type_ids AS entity_type | fields - _* | fields host, entity_type

5. Click the search icon to run the search and confirm that one or more hosts are shown with all columns populated.
6. Click Next.
7. In the Import Column As column, set the host field to Entity Title. Set all other fields to Entity Information Field.
8. Set Conflict Resolution to Update Existing Entities and set the Conflict Resolution Field to host.
9. Click Import.
10. After the import completes, click Set up Recurring Import.
11. Name the recurring import ITSI discovery of Unix and Linux servers and set the frequency based on the needs of your deployment. Use Run on cron Schedule for maximum flexibility.
12. Click Submit.
    ITSI creates the new modular input in $SPLUNK_HOME/etc/apps/itsi/local/inputs.conf.

Tune KPI base searches

This content pack ships with the following KPI base searches:

• OS:Performance.NIX-bandwith
• OS:Performance.NIX-cpu
• OS:Performance.NIX-df
• OS:Performance.NIX-iostat
• OS:Performance.NIX-vmstat

Each search runs every 5 minutes with a 5-minute calculation window and uses only the latest value on a per-entity basis. The 5-minute calculation window ensures that you won't see N/A for less frequent data. Using the latest value means that the KPI status refreshes as quickly as possible for data collected more frequently.

You must review and tune all base searches to run at a frequency that matches your data collection interval.

Tune KPI thresholds

Aggregate KPI thresholds use Normal, Medium, and Low levels, while per-entity thresholds except for available disk space don't exceed the Medium level. Lower threshold levels for OS-level monitoring allow application-level KPIs to take a more prominent threshold level. For example, a server at 100% CPU isn't a critical issue if the apps running on that server are responding normally.

Aggregate threshold values are calculated for general use only. You must tune these threshold values according to your environment. Use the sample service that's linked to the Unix and Linux server health service template to validate your thresholds. For more information, see Overview of creating KPIs in ITSI in the Service Insights manual.