Configure SOAR system logs using Splunk App for SOAR

Splunk App for SOAR includes SOAR system logs, which forwards log files using Splunk Universal Forwarder to an external Splunk instance to create dashboards or use an information-technology service intelligence (ITSI) to monitor the health of you Splunk SOAR (On-premises) environments. The ITSI service contains specific key performance indicators (KPI) for monitoring metrics as well as the operating system.

The ITSI service requires that you install Splunk Add-on for Unix and Linux and configure it to collect and send data to your deployment.

Install Splunk App for SOAR, which includes SOAR system logs, onto both the Splunk SOAR (On-premises) and Splunk Cloud Platform or Splunk Enterprise instance in addition to Splunk Universal Forwarder, even if you don't plan to use the ITSI service.

This article describes logs for the system where Splunk SOAR is installed. For information on logs for data within Splunk SOAR, see Set up remote search on a standalone Splunk Enterprise instance or Set up remote search on a distributed Splunk Enterprise instance or Splunk Cloud Platform instance.

Satisfy the requirements for configuring the SOAR system logs

Follow these instructions to prepare the system for the SOAR system logs.

Before you begin

Configure Splunk SOAR (On-premises) with JSON log format by issuing the following command from the Splunk SOAR console:
$phenv set_preference --logging-format json

Step 1: Install a universal forwarder on each Splunk SOAR server

The universal forwarder collects data from a data source or another forwarder and sends it to a forwarder or a Splunk deployment. You must install a universal forwarder on each Splunk SOAR server you plan to monitor.

1. Install a universal forwarder on each Splunk SOAR server you plan to monitor. For instructions, see Install the universal forwarder software.

   Because each Splunk SOAR server already includes an embedded copy of Splunk Enterprise, the universal forwarder might detect a port conflict during the initial startup. This can adversely affect automated installation scripts. When you install manually, you might be prompted to enter an alternate port. The alternate port is stored in $SPLUNK_HOME/etc/system/local/web.conf.

   Checking prerequisites...
           Checking mgmt port [8089]: not available
   ERROR: mgmt port [8089] - port is already bound.  Splunk needs to use this port.
   Would you like to change ports? [y/n]: y
   Enter a new mgmt port: 8189
   Setting mgmt to port: 8189
   The server's splunkd port has been changed.
           Checking mgmt port [8189]: open
   

2. Configure forwarding on each Splunk SOAR server with outputs.conf. For more information, see Configure forwarders with outputs.conf.

See About forwarding and receiving in the Splunk Enterprise Forwarding Data manual to learn how to install and configure universal forwarders.

Step 2: Deploy Splunk SOAR system logs to indexing and search head tiers

Splunk SOAR contains the following configurations that you must install on the indexing tier and search head:

• Search time extractions and macros in props.conf and macros.conf
• Index time extractions and linebreaking rules in props.conf

Create a Splunk index for Splunk SOAR data before the universal forwarder can send data to them. Navigate to the Configurations tab, open the Create indexes tab, and then select the Create Indexes button.

Install the Splunk App for SOAR to the following locations:

• Any search head
• Indexers
• Any heavy forwarders that the Splunk SOAR server's universal forwarders might send data to
• Universal forwarders that you installed in step 1. See the next section for instructions to configure Splunk SOAR inputs for universal forwarders.

Step 3: Configure Splunk SOAR inputs for universal forwarders

After you install the Splunk App for SOAR on the universal forwarders on your Splunk SOAR instances, make the following changes to enable data collection:

1. Copy the contents of the inputs.conf file from $SPLUNK_HOME/etc/apps/splunk_app_soar/default/ to $SPLUNK_HOME/etc/apps/splunk_app_soar/local/inputs.conf.
2. Set the appropriate index if it's different than the default splunk_app_soar index
3. Optionally, disable any inputs you don't want to collect

Alternatively, if you don't want to deploy the Splunk App for SOAR to your universal forwarders, you can create your own inputs.conf file using the inputs.conf stanzas from the SOAR system logs and placing the resulting inputs.conf file in an appropriate location on your universal forwarder. See the following example file:

Sample inputs.conf

# Phantom Daemon Logs
[monitor:///opt/phantom/var/log/phantom/ingestd.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:daemon
disabled = false

[monitor:///opt/phantom/var/log/phantom/decided.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:daemon
disabled = false

[monitor:///opt/phantom/var/log/phantom/workflowd.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:daemon
disabled = false

[monitor:///opt/phantom/var/log/phantom/actiond.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:daemon
disabled = false

[monitor:///opt/phantom/var/log/phantom/wsgi.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:daemon
#sourcetype = splunk_app_soar:wsgi
disabled = false

[monitor:///opt/phantom/var/log/phantom/watchdogd.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:daemon
disabled = false

[monitor:///opt/phantom/var/log/phantom/proxyd.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:daemon
disabled = false

[monitor:///opt/phantom/var/log/phantom/spawn.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:daemon
disabled = false

############################################################################
# Postgres - log location is the same regardless of Splunk-SOAR type
[monitor:///opt/phantom/data/db/pg_log/*]
index = splunk_app_soar
sourcetype = splunk_app_soar:postgres
disabled = false

############################################################################
# nginx web server - use nginx app on splunkbase for parsing https://splunkbase.splunk.com/app/3258/
# Data is available on both Splunk Search and ITSI. If you configure ITSI, the nginx data is further enhanced 
# with additional observability by ITSI KPI. 
[monitor:///opt/phantom/var/log/nginx/access.log]
index = splunk_app_soar
sourcetype = nginx:plus:access
disabled = false

[monitor:///opt/phantom/var/log/nginx/error.log]
index = splunk_app_soar
sourcetype = nginx:plus:error
disabled = false

############################################################################
# Auditd - use TA-auditd for parsing https://splunkbase.splunk.com/app/4232/
[monitor:///var/log/audit/audit.log]
index = splunk_app_soar
sourcetype = linux:audit
disabled = false

############################################################################
#Log files other than Daemon

[monitor:///opt/phantom/var/log/phantom/*log] #some files have no file types defined
index = splunk_app_soar
sourcetype = splunk_app_soar:misc-logs
blacklist = (spawnd\.log|actiond\.log|decided\.log|ingestd\.log|watchdogd\.log|workflowd\.log|wsgi\.log|proxyd\.log|clusterd\.log)
disabled = false

############################################################################
# Cluster only
[monitor:///opt/phantom/var/log/phantom/clusterd.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:cluster
disabled = false

[monitor:///opt/phantom/var/log/phantom/consul-stderr.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:cluster
disabled = false

[monitor:///opt/phantom/var/log/phantom/consul-stdout.log]
index = splunk_app_soar
sourcetype = splunk_app_soar:cluster
disabled = false


#### SOAR Audit ####

[audit]
start_by_shell = false
python.version = python3
sourcetype = soar
interval = 30
start =
disabled = 0

Step 4: Configure the remote-search service

The remote-search service defines indexes and roles used by Splunk SOAR when it is configured to use an external Splunk Cloud Platform or Enterprise instance for search data. The remote-search service is required in order to use the ITSI service.

The remote search creates various Splunk SOAR indexes using the format phantom_<name>. For example, it creates the index phantom_app_run which ITSI KPIs use.

Follow the instructions to Configure remote search at these locations:

• Search heads
• Indexers

Step 5: Install and configure the Content Pack for Monitoring Unix and Linux

The monitoring approaches in this content pack leverage the Splunk Add-on for Unix and Linux on the universal forwarder. This content pack also requires the full installation and configuration of the Content Pack for Monitoring Unix and Linux.

Before continuing to the next section, complete each of the following installation steps:

1. Deploy the Splunk Add-on for Unix and Linux to indexers and your ITSI search head. For instructions, see Data requirements for the Content Pack for Monitoring Unix and Linux.
2. Install and configure the Content Pack for Monitoring Unix and Linux. For instructions, see Install and configure the Content Pack for Monitoring Unix and Linux.
3. Deploy the Splunk Add-on for Unix and Linux to the universal forwarder on each Splunk SOAR server. Configure an inputs.conf file with the recommended settings for the content pack. For instructions, see Data requirements for the Content Pack for Monitoring Unix and Linux.

Configure ITSI Content Pack for SOAR System Logs

For details on the Content Pack for SOAR System Logs, see About the Content Pack for SOAR System Logs.

Splunk App for SOAR includes the following services:

• Splunk App for SOAR - OSMetrics
• Splunk App for SOAR - System Health

Configure Splunk App for SOAR - OSMetrics

The Splunk App for SOAR - OSMetrics service uses entity filtering to filter entities to it. For more information about entity filtering, see Split and filter a KPI by entities in ITSI in the Service Insights manual. Perform the following steps to create a service for OS monitoring:

1. In ITSI, select Configuration > Services.
2. Open the Splunk App for SOAR - OSMetrics service.
3. Select the Entities tab.
4. In the Alias host matches field, list each of the servers you plan to monitor.
5. Review the list of matched entities and make sure you see one entity for each server.
6. Select Save to save the service configuration.

For more information about configuring entity rules, see Define entity rules for a service in ITSI in the Service Insights manual.

Splunk App for SOAR - System Health

The Splunk App for SOAR - System Health service is for application-level KPIs used to detect system faults and errors, which are reported in system logs.

Tune KPI thresholds

After you configure your Splunk SOAR services, you must tune the thresholds within each Splunk SOAR KPI to meet the specifics of your environment. It's best to do this when you have at least a week of data in your Splunk SOAR environment.

First, review every KPI to determine whether it's one that you think will be helpful in identifying if your Splunk SOAR service is degraded. If the KPI doen't turn out to be a good indicator of service degradation, it's best to remove it to keep your implementation simple. This also makes it easier to find the information that will help lead you to the insight you're looking for.

Review and refine every KPI threshold to ensure the best accuracy of service health scores and creation of notable events. Use the following resources to configure KPI thresholds:

• For regular thresholding instructions, see Configure KPI thresholds in ITSI.
• For more advanced thresholding options, see Overview of advanced thresholding in ITSI.

Configure alerting and notification settings

Configure ITSI to send you alerts when one or more KPIs are experiencing a sustained degradation.

• To generate a notable event when search results meet specific conditions, see Generate events with correlation searches in ITSI.
• To group these notable events into episodes to organize them in Episode Review, see Overview of aggregation policies in ITSI.