Dispatch episode actions to a remote ITSI instance

Dispatch episode actions on-premises from a Splunk Cloud Platform instance using hybrid action dispatching in IT Service Intelligence (ITSI). You can configure your on-premises instance to connect to Splunk Cloud Platform and get the required information to run the action. Actions include updating the status, severity, and owner of episodes, adding comments, linking tickets, sending an email, pinging a host, and any other custom actions you've configured. For all available episode actions, see Configure episode action rules in ITSI.

Hybrid action dispatching involves configuring your cloud instance as the Manager node and your on-premises instance as the Executor node.

Node role	Description
Manager	The node running core Event Analytics functionality. Configure aggregation policies and trigger actions from the Manager node.
Executor	The node where actions run. The Executor node receives actions dispatched from the Manager node and executes them.

The following tasks are meant to configure your cloud instance as the Manager node and your on-premises instance as the Executor node. However, you can configure both roles on on-premises instances if needed.

Prerequisites

• You must have the Splunk admin role to configure hybrid action dispatching.
• If an action is configured on the Manager node, it must also be configured on the Executor node. If there's a mismatch, the Manager node might be able to configure actions that don't exist on the Executor.

Configure the cloud search head as the Manager node

Configure the cloud search head as the Manager node. This is the node running core Event Analytics functionality.

1. Create an account on the Manager node

Configure a user with the itoa_admin role on the cloud search head.

1. Click Settings > Users
2. Click New User.
3. Provide a name and password.
4. In the Available item(s) list, select itoa_admin to add it to the Selected Item(s) list.
5. Click Save.

2. Configure the Manager node

Configure the cloud search head as the Manager node.

1. Navigate back to IT Service Intelligence on the Manager node
2. Click Configuration > Hybrid Action Dispatching.
3. Set the node's role to Manager.
4. Click Save.

3. Disable action execution on the Manager node

The IT Service Intelligence Actions Queue Consumer processes KV store data and executes episode actions. Disable this component on the Manager node so that dispatched actions don't run locally.

The ITSI action queue consumer settings are unreachable on Splunk Cloud Platform. Splunk Cloud Platform customers must work with Splunk Support to disable the action queue consumers.

1. On the Manager node, click Settings > Data inputs.
2. Open the IT Service Intelligence Actions Queue Consumer input.
3. Click Disable in the Status column of all instances to disable them.

4. Configure receiving on the Manager node

Configure the Manager node to receive all action execution information from the Executor node.

1. On the Manager node, click Settings > Forwarding and receiving.
2. Click Configure receiving.
3. Click New Receiving Port.
4. Add the TCP port number of the on-premises instance that will execute actions.
5. Click Save.

Configure the on-premises search head as the Executor node

Configure the on-premises search head as the Executor node. This node executes episode actions. The Executor makes outbound communication on port 8089 to the cloud search head (Manager node), pulling data from the Manager node.

You don't need to open any inbound ports. The Executor pushes data to the Manager node by configuring forwarding on the port you specify.

5. Configure the Executor node

Assign the on-premises search head as the Executor node and configure the remote instance credentials.

1. Within ITSI on the Executor node, click Configuration > Hybrid Action Dispatching.
2. Set the node role to Executor.
3. Configure the following settings:

   Setting	Description
   URI	The location of the Manager node running core Event Analytics services. The URI must point to the management port 8089 (by default) of the Splunk platform instance and include a scheme, host, and port.
   Username	The username that you configured when you created an account on the Manager node.
   Password	The password used to log in to the Manager node.

4. Click Save.
5. Restart the Executor node to point the Action Queue Consumer to the Manager node.

6. Disable the Rules Engine on the Executor node

Disable the Rules Engine on the Executor node so it doesn't run locally.

1. On the Executor node, click Settings > Searches, reports, and alerts.
2. Change the App: context to All.
3. Search for the itsi_event_grouping search. The Rules Engine runs when this search is enabled.
4. In the Actions column, click Edit > Disable to disable the Rules Engine on the Executor node.

7. Configure forwarding on the Executor node

Configure forwarding on the Executor node so that it can send action execution information to the Manager node.

1. On the Executor node, click Settings > Forwarding and receiving.
2. Click Configure forwarding.
3. Click New Forwarding Host.
4. Enter the host and port number of the Manager node.
5. Click Save.

8. Make sure the action queue consumers are running on the Executor node

Perform the following steps on the Executor node:

1. On the Executor node, click Settings > Data inputs.
2. Open the IT Service Intelligence Actions Queue Consumer input.
3. Make sure the alpha, beta, and gamma instances show Enabled in the Status column. If not, enable them.

9. (Optional) Enable additional action queue consumers

ITSI provides five preconfigured action queue consumers with only three enabled by default. If actions show high latency, such as 30 or more seconds to run an action, enable additional action queue consumers. For scaling purposes you can enable additional consumers on a single instance first. If you need additional action throughput, consider scaling out to a second executor node.

The default settings for action queue consumers, such as execution delay time and batch size, can support most ITSI environments. If your environment generates very high throughput, such as 1000 or more actions per minute, consider increasing the batch size for your action queue consumers.

Confirm setup

To confirm that you've successfully configured hybrid action dispatching, execute an action from the Manager node. After the action runs, it should appear in the Activity tab of the episode.

• To execute an action from Episode Review, see Take action on an episode in ITSI.
• To execute an action through an aggregation policy, see Configure episode action rules in ITSI.

Troubleshoot: Why is the Manager node executing actions?

If the Executor node is unreachable for any reason, the Manager node tries to execute actions locally through the REST API instead of queuing the jobs. If instead you prefer that the Rules Engine queues up the actions while waiting for the Executor to become available, you can increase the action consumer refresh rate. By increasing this setting, you increase the amount of time actions can be queued while waiting for the Executor to become available.

Prerequisites

• Only users with file system access, such as system administrators, can configure the action consumer refresh rate.
• Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.

Steps

1. Open or create a local copy of itsi_rules_engine.properties at $SPLUNK_HOME/etc/apps/SA-ITOA/local.
2. Paste the following setting into the file

   action_consumer_refresh_rate = <seconds>
   

3. Set the refresh rate to a period longer than you expect the Executor to be unreachable. For example, if set to 86400 seconds (1 day), the Rules Engine can keep queuing actions for a day before running them.

There are no side effects of this change if the number of accumulated actions doesn't fill up the KV store. However, you might experience long latency for action execution if the Executor remains unavailable for a long time. Because the refresh rate only applies to action consumers, it has no impact on other aspects of ITSI's Event Analytics functionality.