Configure multiple ITSI deployments to use the same indexing layer
You can deploy separate non-clustered IT Service Intelligence (ITSI) search heads for different purposes that forward data to the same indexers. For example, you can use one search head for production and a second search head for testing. You can also deploy separate search head clusters that use the same indexer cluster. In each case, the search heads must be running the same version of ITSI and Splunk Enterprise.
Before configuring multiple ITSI environments to search against the same indexing tier, you must first follow these steps to ensure that different ITSI environments don't end up inadvertently writing to the same ITSI indexes and polluting the results of your production environment.
High-level steps:
- Create new indexes for each ITSI index with the name of the environment appended to the original index name.
- Update index in backfill searches for custom indexes.
- Configure the ITSI search heads to write to the newly created indexes.
- Restart your Splunk software.
- Validate that your new environment is configured to write to the new indexes.
Create new indexes
On each Splunk indexer, create a new index for each of the itsi_*
and anomaly_detection
indexes listed in $SPLUNK_HOME/etc/apps/SA-IndexCreation/default/indexes.conf
. Append the name of the environment to the original index name. Ensure that index names have not changed when you upgrade between ITSI versions.
For example:
itsi_summary
>itsi_summary_dev
itsi_summary_metrics
>itsi_summary_metrics_dev
itsi_tracked_alerts
>itsi_tracked_alerts_dev
itsi_notable_audit
>itsi_notable_audit_dev
itsi_notable_archive
>itsi_notable_archive_dev
itsi_grouped_alerts
>itsi_grouped_alerts_dev
anomaly_detection
>anomaly_detection_dev
itsi_im_metrics
>itsi_im_metrics_dev
itsi_import_objects
>itsi_import_objects_dev
For more information about creating indexes, see Create custom indexes in the Splunk Enterprise Managing Indexers and Clusters of Indexers manual.
Update index in backfill searches for custom indexes
If you are using a custom index, you must change the index name in periodic backfill searches. See Update index in backfill searches for custom indexes in the Event Analytics manual for steps to do so.
Configure search heads to write to the new indexes
Perform one of the following steps depending on your deployment:
Non-clustered
On each search head, create local versions of the following files:
- $SPLUNK_HOME/etc/apps/itsi/local/alert_actions.conf
- $SPLUNK_HOME/etc/apps/itsi/local/savedsearches.conf
- $SPLUNK_HOME/etc/apps/SA-ITOA/local/macros.conf
- $SPLUNK_HOME/etc/apps/SA-ITOA/local/alert_actions.conf
- $SPLUNK_HOME/etc/apps/SA-ITOA/local/itsi_rules_engine.properties
In each file, change the default ITSI index names to the new index names you want to use for the data from that search head. For more information, see How to edit a configuration file in the Splunk Enterprise Admin Manual.
Search head cluster
Make the changes described in the non-clustered steps above on the deployer at etc/shcluster/apps
and push the changes to the cluster members. For more information, see Deploy a search head cluster in the Splunk Enterprise Distributed Search manual.
Restart your Splunk software
Restart your Splunk software or perform a rolling restart to put the changes into effect. For more information, see Restart the search head cluster in the Distributed Search manual.
Validate setup
On each search head, perform the following steps to confirm that searches are pointing to the correct indexes:
- In Splunk Web, navigate to Settings > Data inputs > HTTP Event Collector. Look for the renamed index names for the five ITSI event management tokens with the following source types:
itsi_notable:event
,itsi_notable:archive
,itsi_notable:audit
,itsi_notable:group
. - Check the Event Analytics Audit dashboard to make sure the searches run as expected. For more information, see Event Analytics Audit dashboard in the Event Analytics manual.
- Replace macro searches with the name of the renamed index. For example, the following searches should return the same events:
`itsi_event_management_index_with_close_events` | stats count AS events
index="<new name for itsi_tracked_alerts>" | stats count AS events
- Make sure the data is displaying as expected in service analyzers, deep dives, glass tables, and Episode Review.
- Verify that ITSI users can access the new indexes.
Configure indexes in ITSI | Uninstall Splunk IT Service Intelligence |
This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.17.0, 4.17.1, 4.18.0, 4.18.1, 4.19.0, 4.19.1
Feedback submitted, thanks!