Configure multiple ITSI deployments to use the same indexing layer

You can deploy separate non-clustered IT Service Intelligence (ITSI) search heads for different purposes that forward data to the same indexers. For example, you can use one search head for production and a second search head for testing. You can also deploy separate search head clusters that use the same indexer cluster. In each case, the search heads must be running the same version of ITSI and Splunk Enterprise.

Before configuring multiple ITSI environments to search against the same indexing tier, you must first follow these steps to ensure that different ITSI environments don't end up inadvertently writing to the same ITSI indexes and polluting the results of your production environment.

High-level steps:

1. Create new indexes for each ITSI index with the name of the environment appended to the original index name.
2. Update index in backfill searches for custom indexes.
3. Configure the ITSI search heads to write to the newly created indexes.
4. Restart your Splunk software.
5. Validate that your new environment is configured to write to the new indexes.

Create new indexes

On each Splunk indexer, create a new index for each of the itsi_* and anomaly_detection indexes listed in $SPLUNK_HOME/etc/apps/SA-IndexCreation/default/indexes.conf. Append the name of the environment to the original index name. Ensure that index names have not changed when you upgrade between ITSI versions.

For example:

• itsi_summary > itsi_summary_dev
• itsi_summary_metrics > itsi_summary_metrics_dev
• itsi_tracked_alerts > itsi_tracked_alerts_dev
• itsi_notable_audit > itsi_notable_audit_dev
• itsi_notable_archive > itsi_notable_archive_dev
• itsi_grouped_alerts > itsi_grouped_alerts_dev
• anomaly_detection > anomaly_detection_dev
• itsi_im_metrics > itsi_im_metrics_dev
• itsi_import_objects > itsi_import_objects_dev

For more information about creating indexes, see Create custom indexes in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.

Update index in backfill searches for custom indexes

If you are using a custom index, you must change the index name in periodic backfill searches. See Update index in backfill searches for custom indexes in the Event Analytics manual for steps to do so.

Configure search heads to write to the new indexes

Perform one of the following steps depending on your deployment:

Non-clustered

On each search head, create local versions of the following files:

• $SPLUNK_HOME/etc/apps/itsi/local/alert_actions.conf
• $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf
• $SPLUNK_HOME/etc/apps/SA-ITOA/local/macros.conf
• $SPLUNK_HOME/etc/apps/SA-ITOA/local/alert_actions.conf
• $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties

In each file, change the default ITSI index names to the new index names you want to use for the data from that search head. For more information, see How to edit a configuration file in the Splunk Enterprise Admin Manual.

Search head cluster

Make the changes described in the non-clustered steps above on the deployer at etc/shcluster/apps and push the changes to the cluster members. For more information, see Deploy a search head cluster in the Splunk Enterprise Distributed Search manual.

Restart your Splunk software

Restart your Splunk software or perform a rolling restart to put the changes into effect. For more information, see Restart the search head cluster in the Distributed Search manual.

Validate setup

On each search head, perform the following steps to confirm that searches are pointing to the correct indexes:

1. In Splunk Web, navigate to Settings > Data inputs > HTTP Event Collector. Look for the renamed index names for the five ITSI event management tokens with the following source types: itsi_notable:event, itsi_notable:archive, itsi_notable:audit, itsi_notable:group.
2. Check the Event Analytics Audit dashboard to make sure the searches run as expected. For more information, see Event Analytics Audit dashboard in the Event Analytics manual.
3. Replace macro searches with the name of the renamed index. For example, the following searches should return the same events:

   `itsi_event_management_index_with_close_events` | stats count AS events

   index="<new name for itsi_tracked_alerts>" | stats count AS events

4. Make sure the data is displaying as expected in service analyzers, deep dives, glass tables, and Episode Review.
5. Verify that ITSI users can access the new indexes.