Splunk® IT Service Intelligence

Event Analytics Manual

Download manual as PDF

This documentation does not apply to the most recent version of ITSI. Click here for the latest version.
Download topic as PDF

Dispatch episode actions to a remote ITSI instance

Dispatch episode actions on-premises from a Splunk Cloud instance using hybrid action dispatching in IT Service Intelligence (ITSI0. You can configure your on-premises instance to connect to the cloud and get the required information to run the action. Actions include updating the status, severity, and owner of episodes, adding comments, linking tickets, sending an email, pinging a host, and any other custom actions you've configured.

Hybrid action dispatching involves configuring your cloud instance as the Master node and your on-premises instance as the Executor node.

Node role Description
Master The node running core Event Analytics functionality. Configure aggregation policies and trigger actions from the Master node.
Executor The node where actions run. The Executor node receives actions dispatched from the Master node and executes them.

The following tasks are meant to configure your cloud instance as the Master node and your on-premises instance as the Executor node. However, you can configure both roles on on-premises instances if needed.

Prerequisites

    • You must have the admin or itoa_admin role to configure hybrid action dispatching.
    • If an action is configured on the Master node, it must also be configured on the Executor node. If there's a mismatch, the Master node might be able to configure actions that don't exist on the Executor. For information about configuring action rules, see Configure episode action rules in ITSI.

Configure the cloud search head as the Master node

Configure the cloud search head as the Master node. This is the node running core Event Analytics functionality.

1. Create an account on the Master node

Configure a user with the itoa_admin role on the cloud search head.

  1. Click Settings > Access controls.
  2. In the Users row, click +Add New.
  3. Provide a name and password.
  4. In the Available item(s) list, select itoa_admin to add it to the Selected Item(s) list.
  5. Click Save.

2. Configure the Master node

Configure the cloud search head as the Master node.

  1. On the Master node, click Configuration > Hybrid Action Dispatching.
  2. Set the node's role to Master.
  3. Click Save.

3. Disable action execution on the Master node

The IT Service Intelligence Actions Queue Consumer processes KV store data and executes episode actions. Disable this component on the Master node so that dispatched actions won't run locally.

The ITSI action queue consumer settings are unreachable on Splunk Cloud. Splunk Cloud customers must work with Splunk Support to disable the action queue consumers.

  1. On the Master node, click Settings > Data inputs.
  2. Open the IT Service Intelligence Actions Queue Consumer.
  3. Click Disable in the Status column of the alpha, beta, and gamma instances to disable them.

4. Configure receiving on the Master node

Configure the master node to receive all action execution information from the Executor node.

  1. On the Master node, click Settings > Forwarding and receiving.
  2. Click Configure receiving.
  3. Click New Receiving Port.
  4. Add the TCP port number of the on-premises instance that will execute actions.
  5. Click Save.

Configure the on-premises search head as the Executor node

Configure the on-premises search head as the Executor node. This is the node that executes episode actions. The Executor makes outbound communication on port 8089 to the cloud search head (Master node), pulling data from the Master node.

You do not need to open any inbound ports. The Executor pushes data to the Master node by configuring forwarding on the port you specify.

5. Configure the Executor node

Assign the on-premises search head as the Executor node and configure the remote instance credentials.

  1. On the Executor node, click Configuration > Hybrid Action Dispatching.
  2. Set the node's role to Executor.
  3. Configure the following settings:
    Setting Description
    URI The location of the Master node running core Event Analytics services. The URI must point to the management port 8089 (by default) of the Splunk platform instance and include a scheme, host, and port.
    Username The username that you configured when you created an account on the Master node.
    Password The password used to log in to the Master node.
  4. Click Save.
  5. Restart the Executor node to point the Action Queue Consumer to the Master node.

6. Disable Event Analytics on the Executor node

Disable Event Analytics on the Executor node so that this component does not run locally.

  1. On the Executor node, click Settings > Searches, reports, and alerts.
  2. Change the App: context to All.
  3. Search for the itsi_event_grouping search. ITSI Event Analytics runs when this search is enabled.
  4. In the Actions column, click Edit > Disable to disable Event Analytics on the Executor node.

7. Configure forwarding on the Executor node

Configure forwarding on the Executor node so that it can send action execution information to the Master node.

  1. On the Executor node, click Settings > Forwarding and receiving.
  2. Click Configure forwarding.
  3. Click New Forwarding Host.
  4. Enter the host and port number of the Master node.
  5. Click Save.

8. Ensure that the Action Queue Consumer is running on the Executor node

Make sure that the Event Analytics saved search is running on the Executor node.

  1. On the Executor node, click Settings > Data inputs.
  2. Open the IT Service Intelligence Actions Queue Consumer.
  3. Make sure the alpha, beta, and gamma instances show Enabled in the Status column. If not, enable them.

Confirm setup

To confirm that you've successfully configured hybrid action dispatching, execute an action from the Master node. After the action runs, it should appear in the Activity tab of the episode.

Troubleshoot: Why is the Master node executing actions?

If the Executor node is unreachable for any reason, the Master node tries to execute actions locally through the REST API instead of queuing the jobs. If instead you prefer that the Rules Engine queues up the actions while waiting for the Executor to become available, you can increase the action consumer refresh rate. By increasing this setting, you increase the amount of time actions can be queued while waiting for the Executor to become available.

Prerequisites

  • Only users with file system access, such as system administrators, can configure the action consumer refresh rate.
  • Review the steps in How to edit a configuration file in the Splunk Enterprise Admin Manual.

Never change or copy the configuration files in the default directory. The files in the default directory must remain intact and in their original location. Make changes to the files in the local directory.

Steps

  1. Open or create a local copy of itsi_rules_engine.properties at $SPLUNK_HOME/etc/apps/SA-ITOA/local.
  2. Paste the following setting into the file
    action_consumer_refresh_rate = <seconds>
    
  3. Set the refresh rate to a period longer than you expect the Executor to be unreachable. For example, if set to 86400 seconds (1 day), the Rules Engine can keep queuing actions for a day before running them.

There are no side effects of this change if the number of accumulated actions doesn't fill up the KV store. However, you might experience long latency for action execution if the Executor remains unavailable for a long time. Because the refresh rate only applies to action consumers, it has no impact on other aspects of ITSI's Event Analytics functionality.

Last modified on 14 September, 2020
PREVIOUS
Set up custom episode actions in ITSI
  NEXT
Overview of the ITSI Rules Engine

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.5.0 Cloud only, 4.5.1 Cloud only, 4.6.0 Cloud only, 4.6.1 Cloud only, 4.6.2 Cloud only


Was this documentation topic helpful?

Enter your email address, and someone from the documentation team will respond to you:

Please provide your comments here. Ask a question or make a suggestion.

You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters