Connector module development

Splunk SOAR apps provide connectivity between the Splunk SOAR platform and third party security products and devices. Connector modules are written in Python and imported into Splunk SOAR as Python modules when packaged within an app.

When an action is run, the Splunk SOAR action daemon, actiond, invokes an executable that imports the appropriate module and runs the action. When invoked, an IPC connection is opened between the app and actiond. This communication is handled automatically.

App connectors leverage a Splunk SOAR supplied class called BaseConnector that is the main point of contact between the app connector class and the Splunk SOAR platform.

Apps need to generate results of an action run. This result is abstracted in another class called ActionResult that the app connector uses.

The following code shows some Splunk SOAR classes:

# Phantom imports
import phantom.app as phantom
from phantom.app import BaseConnector
from phantom.app import ActionResult

Action workflow

An action is created in the Splunk SOAR platform either through the UI or through a playbook. Depending upon the action and the asset it is run on, one or more apps are chosen and an action JSON containing the asset configuration and action parameters is created for each app that is run. The app connector Python module is loaded up and its member function '_handle_action' is called with the Action JSON as the parameter. A single instance of execution of the app connector is called an app run. Also a single app run occurs for a single action for example, 'block ip' or 'whois domain'.

Don't use the app connector class with implement _handle_action, use implement handle_action. Doing so allows the BaseConnector to use the action JSON first and it validates the parameters and event before the AppConnector::handle_action is called.

The following code is a snippet of the app JSON containing the metadata describing the configuration and action parameters, for an action called 'whois domain' of a sample whois app.

{
    "appid"       : "776ab991-313e-48e7-bccd-e8c9650c239a",
    "name"      : "WHOIS Sample App",
    "description" : "Query WHOIS",
    "publisher": "Phantom",
    "type": "information service",
    "main_module"    : "samplewhois_connector.py",
    "app_version": "1.0",
    "product_vendor": "Generic",
    "product_name": "Whois Server",
    "product_version_regex": ".*",
    "configuration": {
          "whois_server": {
            "description": "Whois server to use to query",
            "data_type": "string",
            "required": false
          }
    },
    "actions": [
      {
        "action" : "samplewhois domain",
        "description": "Run whois lookup on the given domain",
        "type": "investigate",
        "identifier": "whois_domain",
        "read_only": true,
        "parameters": {
          "domain": {
            "description": "Domain to query",
            "data_type": "string",
            "contains": ["domain name"],
            "primary": true,
            "required": true
          }
        },
        ..
        ..
      }
    ]
}

The following is an example of the action JSON containing the configuration and parameter dictionary, for an action called 'whois domain' that is passed from the Splunk SOAR platform to the app.

{
   "config" : {
      ...
      ...
      "whois_server" : "whois.arin.net"
   },
   "container_id" : "36f4639d-4574-441b-ac39-8a039a5a8534",
   ...
   ...
   "identifier" : "whois_domain",
   "parameters" : [
      {
         "domain" : "amazon.com"
      },
      {
         "domain" : "google.com"
      }
   ]
}

At runtime, configuration and parameter dictionaries are never accessed directly from the app. Call the BaseConnector member functions to get access to these dictionaries.

The parameters key is a list of dictionaries. This is to facilitate multiple actions into one connector run. This means the BaseConnector::_handle_action calls AppConnector::handle_action multiple times, once each with the current parameter dictionary.

{
   "domain" : "amazon.com"
}

Called a second time with the parameter.

{
   "domain" : "google.com"
}

BaseConnector::_handle_action

The following diagram represents the calls that occur between the BaseConnector and the AppConnector for a single connector run.

The app API used by AppConnector is derived from a subclass of BaseConnector. uses Python 3 so the Splunk SOAR ActionD daemon routes to spawn3.

From the top right of the image: BaseConnector::_handle_action validates configuration variables. Error paths are represented by the purple arrows going to the left and down in the image. Once configuration variables are validated, if it is implemented, a call is made to AppConnector::initialize. If there are no errors, then for each entry in the parameter list passed to BaseConnector::_handle_action, a call is made to AppConnector::handle_action, then to AppConnector::finalize, then the connector's run result is generated. If there are exceptions, a call is made to AppConnector::handle_exception , otherwise everything is routed to the ActionD daemon's spawn3 process.

Configuration validation

The BaseConnector::_handle_action parses and loads the app JSON file. This is the file that contains the configuration and parameter variables and the information about which are required. Next, the BaseConnector::_handle_action parses the configuration from 'spawn' and validates it. Before validation it strips all the Python string variable values and deletes the empty values from the configuration dictionary. This way the app writer always gets valid configuration values. In case of an error, the connector result JSON is created and sent back to spawn. If validation succeeds, the AppConnector::initialize() function is called.

AppConnector::initialize()

This function is optional and can be implemented by the AppConnector derived class. Since the configuration dictionary is already validated by the time this function is called, this is a good place to do any extra initialization of internal modules. This function must return a value of either phantom.APP_SUCCESS or phantom.APP_ERROR. If this function returns phantom.APP_ERROR, then the AppConnector::handle_action isn't called. For example, the SMTP app connects to the SMTP server in its initialize function. If this connection fails, then it returns an error and the BaseConnector does not call SmtpConnector::handle_action()

def __init__(self):

    # Call the BaseConnectors init first
    super(SmtpConnector, self).__init__()
    self._smtp_conn = None

def initialize(self):

    self._smtp_conn = None

    try:
        status_code = self._connect_to_server()
    except Exception as e:
        return self.set_status(phantom.APP_ERROR, SMTP_ERR_SMTP_CONNECT_TO_SERVER, e)

    return status_code

Parameter validation

The App JSON contains information about each action including the parameters that it needs and which are required. BaseConnector uses this information to validate each paramater dictionary element. Similar to the configuration dictionary, string values are stripped and if found empty, deleted. If a required key is not present, this error is recorded for the current paramater dictionary and the call to AppConnector::handle_action is skipped.

Parameter 'contains' validation

Every action can specify the contains of an input parameter in the app JSON. The BaseConnector does extra validation based on this contains value. For example, a parameter that has the contains set as ["ip"], will be validated to be a proper IP address. A parameter can have multiple contains specified for example, ["ip", "domain"]. In this case, the parameter is considered valid, even if a single validation passes. No extra validation is carried out if the contains is empty or not specified. If the app doesn't accept the validation done by the BaseConnector, there are two options:

1. Add a validator for a particular contains using the set_validator(..) API
2. Implement the BaseConnector::validate_parameters(...) API, which replaces the complete 'contains' based validation of all the parameters done by the BaseConnector.

AppConnector::handle_action()

This function implements the main functionality of the AppConnector. It is called for every parameter dictionary element in the parameters array. In its simplest form, it gets the current action identifier and then calls a member function of its own to handle the action. This function is expected to create the results of the action run that get added to the connector run. The return value of this function isn't used by the BaseConnector. Instead, it loops over the next parameter element in the parameters array and calls the handle_action again.

AppConnector::finalize()

This function gets called once all the parameter dictionary elements are looped over and no more handle_action calls are left to be made. It gives the AppConnector a chance to loop through all the results that were accumulated by multiple handle_action function calls and create any summary, if required. Another usage is cleanup, disconnecting from remote devices and so on.

AppConnector::handle_exception()

All code within BaseConnector::_handle_action is within a 'try: except:' clause. This makes it so if an exception occurs during the execution of this code it is caught at a single place. The resulting exception object is passed to the AppConnector::handle_exception() to do any cleanup of its own, if required. This exception is then added to the connector run result and passed back to spawn, which gets displayed in the Splunk SOAR UI.

All apps should handle exceptions in their code and return back a proper error message.

Result

As noted previously, each connector run can have multiple internal actions that are carried out. In fact, it can have one action for each element of the parameter list. Each individual action can be represented by an instance of the ActionResult class. In the 'whois domain' sample action, two ActionResult objects are created, one for the whois action carried out for 'amazon.com' and another for 'google.com'. Each of these ActionResults are added to the connector result when the action is carried out. Once all the possible calls to AppConnector::handle_action are complete, the BaseConnector creates the connector run result, in JSOn format, based off all the ActionResult objects that were added and sends it back to spawn.

One thing to note here is that the app writer does not create the result JSON. As actions are performed, ActionResult objects are created, and if they need to be part of the connector run they are added to the connector run using the BaseConnector interface. The ActionResult class contains member functions the app writer uses to set the status, message, and other information about the action. The creation of the result JSON is done automatically by the BaseConnector.

Connector result

A Connector Run result JSON contains the following parts:

• List of action results. Each entry in this list is represented by an ActionResult object that is added by the app author as and when actions are carried out using BaseConnector::add_action_result(...)
• Status of the connector run, success or failure. This value can be set using the BaseConnector::set_status(...) function. In most cases the app doesn't need to set this value explicitly. The BaseConnector loops through the ActionResult list and if all ActionResult objects have their status set to failed, then the connector run is marked as failed. Even if one ActionResult in the list is successful, then the result of the connector run is marked as success.
• Message explaining the connector run result. This value doesn't need to be set by the app. The BaseConnector creates a message containing the number of successful ActionResult objects that are found in the list.
• Summary dictionary. This dictionary is updated by the BaseConnector to contain the total number of actions that ran and how many were successful.

From an app author's point of view, the only part of a connector run that needs to be completed is the ActionResult list.

Action result

Each action result is represented by an object of the ActionResult class. The object encapsulates the following details about an action performed.

Object	Description
Parameter	Every action needs to have a dictionary that represents the parameters that it acted upon and their values. This is required because the input action JSON can have different values than the ones that get operated on by the connector or the connector can add the values of parameters that were not specified. Creating this dictionary allows you to see what values the app used to perform the action. According to the 'whois domain' action the resultant parameter dictionary can be the following: "parameter" : { "domain" : "amazon.com" }, Use ActionResult::update_param(<dict>) to update the parameter dictionary. Other functions that operate on the parameter dictionary are ActionResult::set_param(<dict>) and ActionResult::get_param(<dict>).
Status	Use ActionResult::set_status() to set the status of an ActionResult object. This function also takes an optional message that can be used to set the message of an ActionResult object.
Message	Use ActionResult::set_status() or ActionResult::append_to_message() functions to update the message of an ActionResult. If the app author does not set the message in an ActionResult object, then the object will create a textual representation of the ActionResult summary dictionary.
Summary	This dictionary contains a summary about the action performed. For the 'whois domain' example this dictionary contains the most interested data about the domain. "summary" : { "city" : "Reno", "country" : "US", "organization" : "Amazon Technologies, Inc." }
Data	Data is where the whole output of the action performed is added. This always is a list, even when it will contain only a single item, it is still a list. Use ActionResult::add_data to add the action output to the ActionResult. For the 'whois domain' example the 'data' looks like the following sample: "data" : [ { "contacts" : { "admin" : { "city" : "Reno", "country" : "US", "organization" : "Amazon Technologies, Inc.", ... }, "billing" : null, "registrant" : { "city" : "Reno", "country" : "US", ... }, .... "nameservers" : [ "ns2.p31.dynect.net", "pdns1.ultradns.net" .... ], "raw" : [ "Domain Name: amazon.com\nRegistry Domain ID: 281209_DOMAIN_COM-VRSN\nRegistra ....... ], "registrar" : [ "MarkMonitor, Inc." ], "status" : [ "clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)", ..... ], "updated_date" : [ "2014-04-30T12:11:08" ], "whois_server" : [ "whois.markmonitor.com" ] } .... ]

Output metadata

The output of the action can generate information which is added to the data section of the ActionResult object through the ActionResult::add_data(...) function. The 'output' key in the app JSON file describes the content of this data section. Add this key for every action.

The Splunk SOAR platform parses this section and auto generates the documentation so a playbook writer gets all the required information about the output generated by the action. Rendering this data as a table like widget makes for a better view. This can be done easily by filling up certain keys in the 'output' section. The Splunk SOAR platform matches the output of one action to the input parameter of another by matching the 'contains' keyword. This section allows the app author to specify the 'contains' of the data

See Metadata for more information.

App installation

All the required files of an app including the .json and .pyc files need to be placed in a TAR file. The TAR file contains a single directory containing all app files. For example, to create the installer TAR file of an app called samplewhois use the following command:

tar -zcvf samplewhois.tgz samplewhois

This TAR file can be used to install the app on a Splunk SOAR box using the install app button found on the apps page.

Installed apps are placed in a sub folder created in the apps directory based on the app name and GUID following this format: /opt/phantom/apps/appname_<GUID>/. For example, the samplewhois app is installed in /opt/phantom/apps/whoissample_<GUID>/.

PYTHONPATH

As mentioned in previous topics, the connector modules are called into by the spawn executable for every action carried out. Before spawn calls into various connector functions, it sets up the PYTHONPATH, which includes the Splunk SOAR library directory and the app installed directory. For the samplewhois example the path /opt/phantom/apps/whoissample_<GUID>/ folder will be part of the PYTHONPATH. This allows the app author to distribute modules that are internally used in the app with the app TAR file itself. As long as the app directory TAR filecontains an '__init__.py' ( or __init__.pyc) file distributed with it, the app can safely import modules it has distributed with the app. For example, you can place all the utility/helper functions within a set of files and distribute them with the app. The main connector file can then safely import them at runtime.

CA bundles

It's common for apps to use a REST API to communicate with an external device and use the Python requests module to do so. The requests module picks up the CA bundle file pointed to by the REQUESTS_CA_BUNDLE environment variable. comes with a CA bundle preinstalled. The bundle file is located in your instance at /opt/phantom/etc/cacerts.pem. At runtime for every action that is executed, will set the the REQUESTS_CA_BUNDLE variable to the /opt/phantom/etc/cacerts.pem file before executing the action, so if the app is using the requests module, the CA Bundle will already be set for it to use.

Test connectivity

The Splunk SOAR platform allows the user to configure an asset based on the asset configuration defined by an app in the configuration section of the app metadata. The same UI also allows you to test the configuration. This allows the user to see if the configuration is correct or not. This is implemented by the Splunk SOAR UI through a Test Connectivity button on the asset configuration page and by mapping this button to a 'test_asset_connectivity' action in the app. If the app does not implement this action, the TEST CONNECTIVITY button is not displayed. An app can implement this action by adding the following action into its app JSON.

{
  "action": "test connectivity",
  "description": "Validate the asset configuration for connectivity",
  "verbose": "This action logs into the device using a REST Api call to check validate the asset configuration",
  "type": "test",
  "identifier": "test_asset_connectivity",
  "read_only": true,
  "parameters": {
  },
  "output": [],
  "versions":"EQ(*)"
}

Modify the verbose value to specify how the test connectivity is carried out. This action isn't going to be passed any parameters. It needs to work on the asset configuration only. The identifier key used in the previous example is test_asset_connectivity and must be implemented in the connector script. The action type is set to test. This needs to be the case since the UI checks for this value. Also the progress and results of the action are displayed in a dialog box synchronously so it is helpful to be a bit more descriptive by using the self.save_progress() or self.send_progress() function calls.

Ingestion

In Splunk SOAR, data sources are services or devices that supply information that Splunk SOAR users may want to store or act on. An app can support extracting such data from a device and ingesting it into Splunk SOAR by implementing the on poll action. An app's ingestion action handler can be called in two instances:

1. The Poll Now button of an asset configuration page.
2. Scheduled ingestion.

In each of the previously mentioned instances the parameters that are passed to the action and the action name identifier that is used are described in the following JSON.

If an app does not add this action to the app JSON, ingestion configuration and interactions like the Poll Now button aren't displayed on the corresponding asset.

{
  "action": "on poll",
  "description": "Callback action for the on_poll ingest functionality.",
  "type": "ingest",
  "identifier": "on_poll",
  "read_only": true,
  "parameters": {
    "container_id": {
      "data_type": "string",
      "order": 0,
      "description": "Container IDs to limit the ingestion to.",
      "allow_list": true
    },
    "start_time": {
      "data_type": "numeric",
      "order": 1,
      "description": "Start of time range, in epoch time (milliseconds)",
      "verbose": "If not specified, the default is past 10 days"
    },
    "end_time": {
      "data_type": "numeric",
      "order": 2,
      "description": "End of time range, in epoch time (milliseconds)",
      "verbose": "If not specified, the default is now"
    },
    "container_count": {
      "data_type": "numeric",
      "order": 3,
      "description": "Maximum number of container records to query for."
    },
    "artifact_count": {
      "data_type": "numeric",
      "order": 4,
      "description": "Maximum number of artifact records to query for."
    }
  },
  "output": [],
  "versions":"EQ(*)"
}