Installation and configuration considerations and issues

This topic contains important considerations and issues that you should be aware of when installing and configuring Splunk IT Service Intelligence (ITSI). This list supplements (and in some cases overlaps) with known issues found in the official ITSI release notes. For more information, see Known issues in Release Notes.

Installation/Upgrade

Ubuntu

When installing ITSI on Ubuntu, use Bash shell. Do not use Dash shell.

• Splunk assumes Bash shell.
• Using Dash shell can result in zombie processes.

Windows

When installing ITSI on Windows, rename inputs.conf.windows to inputs.conf, in both SA-Utils and SA-ThreatIntelligence default directories.

For example:

cd $SPLUNK_HOME/etc/apps/SA-Utils/default/
cp inputs.conf inputs.conf.bak
cp inputs.conf.windows inputs.conf
rm inputs.conf.windows

Run a script to set the default team to Global

The new Teams feature introduced in IT Service Intelligence version 3.0 requires that all services be assigned to a team. When upgrading to ITSI 3.0 or later from a version prior to 3.0, the migration script creates the default Global team in the KV store and assigns all existing services to the Global team.

If this step fails, you';ll receive the error "Failed to import Team settings" and ITSI will be unusable. If you encounter this error when upgrading to version 3.0, download and install ITSI version 3.0.1. ITSI 3.0.1 contains a Python script called itsi_reset_default_team.py in the apps/SA-ITOA/bin directory. Running this script manually creates the Global team in the KV store so the migration is successful.

If you encounter this error when upgrading to version 3.0.1 from a version prior to 3.0, manually run the Python script included in version 3.0.1.

Steps

1. Run the following command on any search head in your ITSI deployment:

   cd $SPLUNK_HOME/etc/apps/SA-ITOA/bin
   $SPLUNK_HOME/bin/splunk cmd python itsi_reset_default_team.py
   

2. Provide the splunkd port number and Splunk user name and password when prompted.
3. After the script has successfully finished, the Global group is created in the kvstore.
4. Restart the Splunk platform.

It is best to install version 3.0.1 rather than downloading and running the itsi_reset_default_team.py script from the 3.0.1 installation package. Version 3.0.1 has better error logging than version 3.0. If the script doesn't work, version 3.0.1 will provide more information on the reason for the failure.

Service Analyzer

Real-time searches might hang when the number of KPIs/Services is set too high.

• You can change the number of tiles shown on the Service Analyzer dashboard. If you set the number of tiles too high (50 or greater) this can cause the two indexed real time searches that generate the tiles to hang and show a "Waiting for data" message. This occurs only on the specific search head.
• This seems to occur as a result of KV Store performance issues in a search head cluster environment.

Workarounds:

• Avoid changing the number of tiles setting in a search head cluster environment.
• Use filters to display only the specific services and KPIs that require monitoring.
• Set the number of visible tiles to the lowest number possible.

Data backfill on Windows

On Windows, when using data backfill, ITSI might not work as expected and an error message might appear due to insufficient page file quota allocation.

Workaround: Double the current page file quota allocation on your Windows machine.

Duplicate entity aliases

Duplicate entity aliases occur when you create entities separately, but they happen to have overlapping aliases. Entity aliases must uniquely identify the entity. For example, host=appserver-01. The same alias field value cannot be used for more than one entity. If more than one entity is using the same alias field value (such as appserver-01), it can lead to a miscalculated service health score value (because extra entities are counted) and incorrect KPI base search results.

How to detect duplicate aliases

Use any of the following methods to detect duplicate entity aliases:

• If Splunk finds any duplicates, a warning message appears in the Messages menu. Click Show duplicates to open the ITSI Health Check dashboard which lists the entities with duplicate aliases.
  
• Navigate directly to the ITSI Health Check dashboard by clicking Search > Dashboards > ITSI Health Check from the ITSI main menu.
• Scan your entities manually to check for duplicates.
• Run the following search to detect duplicates:

  | inputlookup itsi_entities | eval original='identifier.values' | mvexpand original | eval key=_key | stats count values(identifier.values) AS entity_aliases values(title) AS entity_title values(key) AS entity_key values(services._key) AS service_keys by original | eval error=if(count>1,"dupe","") | where count>1

What causes duplicate aliases?

The following issues can lead to duplicate entity aliases:

• The same entity was imported several times with different methods and fields.
• An entity was imported by a specific module, but each time with a slightly different name as the title or alias.
• An entity was manually imported from a search or CSV file with different field values, or a case difference.

Clean up and avoid duplicates

Take the following steps to fix duplicate entity aliases and prevent future ones:

Always back up ITSI before attempting to fix duplicates (Configure > Backup/Restore).

• If the duplicates were caused by an automatic import, disable the import in Settings > Data inputs > IT Service Intelligence Asynchronous CSV Loader, then disable the appropriate inputs. Retry the import after you clean up the entities and normalize fields and aliases.
• Click Configure > Entities and edit the entity definitions with duplicate aliases. Keep the alias value for one of the entities and edit the other to remove the duplicate alias value.
• Merge the duplicates by moving all the fields that differ to one entity, then deleting the extra copy.
• Test your entity import searches to ensure there are no conflicts.

Services (or other objects) missing after upgrade

When the objects in ITSI are exported during a backup or migration, if the number of KPIs linked to a service is high, the instance can hit a kvstore memory size limit causing some objects to be dropped from the backup and lost after the upgrade.

Workaround: Increase the KV store bulk get limit in $SPLUNK_HOME/etc/apps/SA-ITOA/local/limits.conf and retry the backup or upgrade. Increase the max_size_per_result_mb value as necessary. The default limit is set to 500MB in ITSI version 3.1.4 and later.

[kvstore]
# The maximum size, in megabytes (MB), of the result that will be returned for a single query to a collection.
# ITSI requires approximately 50MB per 1,000 KPIs. Override this value if necessary.
# Default: 500 MB
max_size_per_result_mb = 500

This action will increase the memory used by the kvstore during operations.