Splunk® Phantom (Legacy)

Administer Splunk Phantom

Splunk Phantom 4.10.7 is the final release of Splunk's Security Orchestration, Automation, and Response (SOAR) system to be called Splunk Phantom. All later versions are named Splunk SOAR (On-premises). For more information, see the Splunk SOAR (On-premises) documentation.

Configure the logging levels for Splunk Phantom daemons

You can adjust logging levels for each daemon running in Splunk Phantom to help debug or troubleshoot issues.

Splunk Phantom daemons

The following daemons in Splunk Phantom work to control collection and scheduling tasks in the background independently from the Splunk Phantom web interface:

Daemon Description
Action daemon Responsible for launching actions by putting into effect the appropriate app on the specified asset. Also responsible for the debug log that says what version of Python is being used. The debug log for Python 3 shows Running executable: spawn3.

The following key actions are logged by this daemon:

  • Manual actions run against any configured asset
  • Scheduled actions against any configured asset
Cluster daemon Responsible for communicating with cluster nodes. Available only in clustered environments.
Decide daemon Responsible for operating on incoming data.

The following key actions are logged by this daemon:

  • Launching active playbooks against new containers for associated Operates On types
  • Playbook validation
  • Playbook loading and use
  • Custom automation or playbook APIs
  • Prompting the action daemon for action and app use
  • Prompting the workflow daemon for the approval workflow process
  • Processing approval response and results from the workflow daemon
  • Matching app execution to specific action results
  • Playbook debugging
  • Counting licensed action uses
Ingest daemon Responsible for ingesting data into the product.

The following key actions are logged by this daemon:

  • Ingestions from data sources that use polling to get new data
  • Asset health reporting, also known as, connectivity checking for the Asset Health dashboard component
  • Configuration changes to any assets or any app-specific configuration
  • Manual Test Connectivity actions launched directly from any asset
Proxy daemon Responsible for communicating with Splunk mobile apps to register devices and send notifications to mobile users. This daemon is available only when the mobile app feature is enabled.
Watchdog daemon Responsible for tracking the status of other daemons and adding or removing them in the system startup list.

The following key actions are logged by the watchdog daemon:

  • Installation of new apps
  • Health monitoring the Splunk Phantom deployment
  • Maintenance of other Splunk Phantom platform daemons and components
  • Restarts of the Splunk Phantom platform daemons and components
Workflow daemon Responsible for managing approval requests to action reviewers and asset owners.

The following key actions are logged by the workflow daemon:

  • Processing and launching approval processes and managing approval escalations
  • Sending user email notifications for container assignment, expiry, manual action requests, and other email templates

Configure the logging level for each Splunk Phantom daemon

Adjust the logging levels as needed to assist Splunk Phantom Support with troubleshooting any issues you might experience.

  1. From the main menu, select Administration.
  2. Select System Health > Debugging.
  3. Select a logging level for each daemon you want to change. The log levels determine the message types that are written to each daemon's corresponding log file. The Debug level is the most verbose level of logging and is useful for troubleshooting.
  4. (Optional) Click Download Logs to download a copy of the current log files for manual investigation or a submission to support.
    A zipped TAR archive of the logs is downloaded to /var/log/phantom.
  5. Click Save Changes.

At the operating system level, log files are in /var/log/phantom. In a non-root installation, the path is ${phantom_home}/var/log/phantom. Adjust logging levels only at the direction of Splunk Phantom Support.

Example log structure

See the following sample of a common log format: 

Oct  5 22:55:18 localhost DECIDED[7177]: TID:7422 : WARNING: DECIDED : rules_engine.cpp : 1503 : DECIDED_CMD_PROCESS_CONTAINERS : All rules FAILED t

This table summarizes the structure of the example log message.

Log message content Description
Oct 5 22:55:18 Timestamp of when the log message was generated.
localhost Name of the host where the log message was generated.
DECIDED[7177]: Name of the component and process ID (PID) generating the message.
TID:7422: Thread ID (TID) of the message.
WARNING: Log level or class of the message.
DECIDED: Functional component that generated the log message.
rules_engine.cpp: Source file applicable to the log message.
1503: Line number in the source file that caused this log message to be generated.
DECIDED_CMD_PROCESS_CONTAINERS: All rules FAILED to process the container: 2964. Error: Playbook 'local/test11 (version: 1, id: 711)' cannot be executed since it is: NOT ACTIVE, ENABLED and VALID The log message.

Monitor Splunk Phantom logs using an external SIEM

You can use an external SIEM such as Splunk Enterprise to monitor the status of your Splunk Phantom instance using the log files from Splunk Phantom. Monitor errors in actions execution through the ingestion of the actiond.log file, and you can find platform information in the watchdogd.log file.

These files are constantly updated by the components of the Splunk Phantom platform. Use a service to monitor the log file regularly, or copy and forward the log files on a regular basis.

Rotate your Splunk Phantom logs when they get too large

The Splunk Phantom logs in /var/log/phantom are automatically rotated when they reach 50MB in size. Splunk Phantom keeps a running archive of the last 10 rotated logs. The oldest log is deleted when a new log is rotated in. You can configure log rotation settings for your organization's IT requirements by modifying the phantom_logrotate.conf file and restarting rsyslog for the changes to take effect.

Perform the following tasks to configure log rotation on Splunk Phantom:

  1. Run the logrotate --version command to find your Logrotate version. For example: 
    [root@phantom]# logrotate --version
logrotate 3.8.6
  2. Configure logrotate using the instructions based on your Logrotate version.
    • If you are running Logrotate version 3.8.x, use a configuration with the su directive. By default, the configuration file is located in /etc/logrotate.d/phantom_logrotate.conf. For example: 
      # Note: If you change size to anything approaching or greater than 100MB
# you need to updated the rsyslog.conf as well as he has a maxsize
# of 100MB which triggers logrotate.
/var/log/phantom/actiond.log
/var/log/phantom/clusterd.log
/var/log/phantom/decided.log
/var/log/phantom/ingestd.log
/var/log/phantom/proxyd.log
/var/log/phantom/watchdogd.log
/var/log/phantom/workflowd.log
/var/log/phantom/spawn.log  {
  su phantom phantom
  rotate 10
  size 50M
  start 1
  missingok
  create 0660 phantom phantom
  postrotate
    kill -HUP $(cat /var/run/syslogd.pid)
  endscript
}

# wsgi log is generated by wsgi itself.
# use copytruncate to avoid needing to restart uwsgi
#
/var/log/phantom/wsgi.log {
  su nginx phantom
  copytruncate
  rotate 10
  size 50M
  start 1
  create 0660 nginx phantom
}


# supervisord generated logs. these applications
# log to stdout/stderr which supervisord directs to files.
#
/var/log/phantom/supervisord.log
/var/log/phantom/-stderr.log
/var/log/phantom/-stdout.log {
  su phantom phantom
  copytruncate
  missingok
  rotate 10
  size 50M
  start 1
  create 0660 phantom phantom
}


# rsync logs are typically generated during warm-standby operation
# on the primary system. they are generated by the rsync tool itself.
/var/log/phantom/rsync*.log {
  su root root
  rotate 10
  size 10M
  start 1
  missingok
  create 0644 root root
}
    • If you are running a Logrotate version older than 3.8.x, the default location and content of the Logrotate configuration file is /etc/logrotate.d/phantom_logrotate.conf. For example: 
      [root@phantom]# more /etc/logrotate.d/phantom_logrotate.conf 
/var/log/phantom/*.log {
  rotate 10
  size 10M
  start 1
  create 0660 root phantom
  postrotate
    kill -HUP $(cat /var/run/syslogd.pid)
  endscript
}
  3. After configuring Logrotate in the configuration file, restart rsyslog using the /opt/phantom/bin/phsvc restart rsyslog command. On a privileged instance of Splunk Phantom: 
    [root@phantom]# /opt/phantom/bin/phsvc restart rsyslog
Shutting down system logger:                               [  OK  ]
Starting system logger:                                    [  OK  ]

    On an unprivileged instance of Splunk Phantom: 

    service rsyslog restart
Redirecting to /bin/systemctl restart rsyslog.service
Last modified on 25 February, 2022
View ingested container statistics using Ingestion Status   Enable and download audit trail logs in Splunk Phantom

This documentation applies to the following versions of Splunk® Phantom (Legacy): 4.9, 4.10, 4.10.1, 4.10.2, 4.10.3, 4.10.4, 4.10.6, 4.10.7

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters