Splunk® IT Service Intelligence

Install and Upgrade Manual

This documentation does not apply to the most recent version of Splunk® IT Service Intelligence. For documentation on the most recent version, go to the latest release.

Steps to address the Apache Log4j vulnerabilities in ITSI or IT Essentials Work

On Friday December 10, 2021, a serious remote code execution (RCE) vulnerability, commonly known as Log4Shell, was discovered in the popular open-source Apache Log4j (versions 2.0 to 2.14.1) logging library. Over subsequent days, additional vulnerabilities have been discovered. See Apache Log4j 2 in Apache documentation for more info. The Apache Software Foundation released a series of emergency patches for these vulnerabilities. For more information on addressing the vulnerabilities, see Splunk Security Advisory for Apache Log4j (CVE-2021-44228 and CVE-2021-45046).

Impacted version Immediate workaround Intermediate upgrade
(Skip unless already in progress) 		Final fix
ITSI and ITE Work versions 4.11.0, 4.9.x (on-premises and cloud)

ITSI 4.7.x (on-premises and cloud)

See the workaround steps provided on this page after this table. Follow these steps in your existing ITSI installation to reduce your exposure to the CVE-2021-44228 vulnerability.


Cloud customers: Splunk Cloud TechOps is upgrading impacted versions.

Upgrade to the maintenance version that includes the fix for CVE-2021-44228:
  • 4.7.x > 4.7.3
  • 4.9.x > 4.9.5
  • 4.11.0 > 4.11.1

See Version-specific upgrade notes for ITSI for steps to take after you upgrade.

Upgrade to the maintenance version that includes version 2.16.0 or later of Apache Log4j libraries, which addresses both CVE-2021-44228 and CVE-2021-45046.
  • 4.7.x > 4.7.4
  • 4.9.x > 4.9.6
  • 4.11.x > 4.11.3

See Version-specific upgrade notes for ITSI for steps to take after you upgrade.

ITSI and ITE Work 4.10.x - Cloud-only version

ITSI 4.5.x, 4.6.x, and 4.8.x - Cloud-only versions

Splunk Cloud TechOps is upgrading impacted versions. All cloud stacks will be upgraded to the closest latest minor version with the fix for CVE-2021-44228.

4.5.x and 4.8.x will be upgraded to 4.9.5.

All cloud stacks will be updated to the closest latest minor version with the 2.16.0 or later version of Apache Log4j.

4.5.x and 4.8.x will be upgraded to 4.9.6.
ITSI version 4.4.x (No longer supported as of October 22, 2021) See the workaround steps provided on this page after this table. Follow these steps in your existing ITSI installation to reduce your exposure to the CVE-2021-44228 vulnerability.
 Upgrade to the 4.7.3. maintenance version that includes the fix for the RCE vulnerability (CVE-2021-44228).


See Version-specific upgrade notes for ITSI for steps to take after you upgrade.

Upgrade to the 4.7.4 maintenance version that includes version 2.16.0 or later of Apache Log4j libraries, which addresses both CVE-2021-44228 and CVE-2021-45046.


See Version-specific upgrade notes for ITSI for steps to take after you upgrade.


Workaround steps for self-managed deployments on standalone search heads on *nix

Follow these steps to implement a workaround for CVE-2021-44228 in self-managed deployments of Splunk IT Service Intelligence or IT Essentials Work on standalone search heads on *nix environments. You can use this fix as a workaround until you have time to upgrade to a maintenance version that addresses the vulnerability. These changes will not disrupt any user functionality. These changes require a maintenance window of about eight to ten minutes.

Prerequisites

  • You must have administrative access to the operating system on the machine where ITSI is installed to perform these steps.

For all search heads where ITSI or ITE Work is installed, perform the following procedures.

$SPLUNK_HOME is the directory where you installed Splunk Enterprise.

Steps

  1. Open a shell prompt.
  2. Go to the folder $SPLUNK_HOME/bin.
  3. Stop the Splunk process. 
    cd $SPLUNK_HOME/opt/splunk/bin
./splunk stop
  4. Go to the etc/apps/SA-ITOA/lib/java/event_management/libs/ directory:
    cd $SPLUNK_HOME/etc/apps/SA-ITOA/lib/java/event_management/libs/
  5. Run the command that corresponds to your version of ITSI or IT Essentials Work to delete the JndiLookup classes from the log4j jar file.
    • ITSI or IT Essentials Work version 4.7.x, 4.9.x, or 4.11.x: 
      zip -q -d log4j-core-2.13.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.13.2 in this directory, you can safely delete these .jar files with lower versions.

    • ITSI version 4.4.x: 
      zip -q -d log4j-core-2.5.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.5 in this directory, you can safely delete these .jar files with lower versions.

  6. Go to the etc/apps/SA-ITSI-MetricAD/lib/ directory. 
    cd $SPLUNK_HOME/etc/apps/SA-ITSI-MetricAD/lib/
  7. Run the command that corresponds to your version of ITSI or IT Essentials Work to delete JndiLookup classes from the log4j jar file.
    • ITSI version 4.7.x, 4.9.x, 4.11.0: 
      zip -q -d org.apache.logging.log4j.log4j-core-2.13.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.13.2 in this directory, you can safely delete these .jar files with lower versions.

    • ITSI version 4.4.x: 
      zip -q -d org.apache.logging.log4j.log4j-core-2.3.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.3 in this directory, you can safely delete these .jar files with lower versions.

  8. Delete older versions of log4j .jar files from the search head. To view a list of the files and their locations, see Version-specific upgrade notes for ITSI.
  9. Restart Splunk Enterprise: 
    cd $SPLUNK_HOME/opt/splunk/bin
./splunk start

Workaround steps for self-managed deployments on standalone search heads on Windows

Follow these steps to implement a workaround for CVE-2021-44228 in self-managed deployments of Splunk IT Service Intelligence or IT Essentials Work on standalone search heads on Windows environments. You can use this fix as a workaround until you have time to upgrade to a maintenance version that addresses the vulnerability. These changes will not disrupt any user functionality. These changes require a maintenance window of about eight to ten minutes.

Prerequisites

  • You must have administrative access to the operating system on the machine where ITSI is installed to perform these steps.
  • You need a zip utility installed on Windows. The instructions below provide the command prompt syntax for 7zip.

For all search heads where ITSI or ITE Work is installed, perform the following procedure.

$SPLUNK_HOME is the directory where you installed Splunk Enterprise.

Steps

  1. Use the Services control panel to stop the Splunk process.
  2. Go to the etc/apps/SA-ITOA/lib/java/event_management/libs/ directory: 
    cd $SPLUNK_HOME\etc\apps\SA-ITOA\lib\java\event_management\libs
  3. Run the command that corresponds to your version of ITSI or IT Essentials Work to delete the JndiLookup classes from the log4j jar file.
    • ITSI or IT Essentials Work version 4.7.x, 4.9.x, or 4.11.x: 
      $SPLUNK_HOME\etc\apps\SA-ITOA\lib\java\event_management\libs>"C:\Program Files\7-Zip\7z.exe" D .\log4j-core-2.13.2.jar org\apache\logging\log4j\core\lookup\JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.13.2 in this directory, you can safely delete these .jar files with lower versions.

    • ITSI version 4.4.x: 
      $SPLUNK_HOME\etc\apps\SA-ITOA\lib\java\event_management\libs>"C:\Program Files\7-Zip\7z.exe" D .\log4j-core-2.5.jar org\apache\logging\log4j\core\lookup\JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.5 in this directory, you can safely delete these .jar files with lower versions.

  4. Go to the etc/apps/SA-ITSI-MetricAD/lib/ directory. 
    cd $SPLUNK_HOME\etc\apps\SA-ITSI-MetricAD\lib\
  5. Run the command that corresponds to your version of ITSI or IT Essentials Work to delete JndiLookup classes from the log4j jar file.
    • ITSI version 4.7.x, 4.9.x, 4.11.0: 
      $SPLUNK_HOME\etc\apps\SA-ITSI-MetricAD\lib\>"C:\Program Files\7-Zip\7z.exe" D .\org.apache.logging.log4j.log4j-core-2.13.2.jar org\apache\logging\log4j\core\lookup\JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.13.2 in this directory, you can safely delete these .jar files with lower versions.

    • ITSI version 4.4.x: 
      $SPLUNK_HOME\etc\apps\SA-ITSI-MetricAD\lib\>"C:\Program Files\7-Zip\7z.exe" D .\org.apache.logging.log4j.log4j-core-2.3.jar org\apache\logging\log4j\core\lookup\JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.3 in this directory, you can safely delete these .jar files with lower versions.

  6. Delete older versions of log4j .jar files from the search head. To view a list of the files and their locations, see Version-specific upgrade notes for ITSI.
  7. Use the Services control panel to restart Splunk Enterprise.

Workaround steps for self-managed deployments on a search head cluster on *nix

Follow these steps to implement a workaround for CVE-2021-44228 in self-managed deployments of Splunk IT Service Intelligence or IT Essentials Work on search head clusters on *nix environments. You can use this fix as a workaround until you have time to upgrade to a maintenance version that addresses the vulnerability. These changes require a maintenance window of about eight to ten minutes.

Prerequisite
You must have administrative access to the operating system on the machine where ITSI or IT Essentials Work is installed to perform these steps.

For all search head clusters where ITSI or ITE Work is installed, perform the following procedures.

$SPLUNK_HOME is the directory where you installed Splunk Enterprise.

Steps

  1. Log on to the deployer.
  2. Go to the etc/shcluster/apps/SA-ITOA/lib/java/event_management/libs/ directory. 
    cd $SPLUNK_HOME/etc/shcluster/apps/SA-ITOA/lib/java/event_management/libs/
  3. Run the command that corresponds to your version of ITSI or IT Essentials Work to delete the JndiLookup classes from the log4j jar file.
    • ITSI or IT Essentials Work version 4.7.x, 4.9.x, or 4.11.x: 
      zip -q -d log4j-core-2.13.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
    • ITSI version 4.4.x: 
      zip -q -d log4j-core-2.5.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
  4. Make a note of the updated jar's checksum.
    • ITSI or IT Essentials Work version 4.7.x, 4.9.x, or 4.11.x:
      On *nix: 
      sha256sum log4j-core-2.13.2.jar
    • ITSI version 4.4.x: 
      sha256sum log4j-core-2.5.jar
  5. Go to the etc/shcluster/apps/SA-ITSI-MetricAD/lib/ directory. 
    cd $SPLUNK_HOME/etc/shcluster/apps/SA-ITSI-MetricAD/lib/
  6. Run the command that corresponds to your version of ITSI or IT Essentials Work to delete JndiLookup classes from the log4j jar file.
    • ITSI version 4.7.x, 4.9.x, 4.11.0: 
      zip -q -d org.apache.logging.log4j.log4j-core-2.13.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
    • ITSI version 4.4.x: 
      zip -q -d org.apache.logging.log4j.log4j-core-2.3.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
  7. Make note of the updated jar's checksum
    • ITSI version 4.7.x, 4.9.x, 4.11.0: 
      sha256sum  org.apache.logging.log4j.log4j-core-2.13.2.jar
    • ITSI version 4.4.x: 
      sha256sum  org.apache.logging.log4j.log4j-core-2.3.jar
  8. Delete the older versions of log4j .jar files from the deployer before deploying the updated jars. To view a list of the files and their locations, see Version-specific upgrade notes for ITSI.
  9. Deploy the updated jars to search heads with the following command. The -target parameter specifies the URI and management port for any member of the cluster. The -auth parameter specifies credentials for the deployer instance. 
    $SPLUNK_HOME/bin/splunk apply shcluster-bundle -target <URI>:<management_port> -auth <username>:<password>
  10. Wait for deployment and rolling restart to complete.
  11. Log on to the search heads and verify the updated jar's checksums.
    1. Go to the etc/apps/SA-ITOA/lib/java/event_management/libs/ directory. 
      cd $SPLUNK_HOME/etc/apps/SA-ITOA/lib/java/event_management/libs/
    2. Verify that the checksum of the jar matches the checksum you made note of earlier.
      • ITSI or IT Essentials Work version 4.7.x, 4.9.x, or 4.11.x: 
        sha256sum log4j-core-2.13.2.jar
      • ITSI version 4.4.x: 
        sha256sum log4j-core-2.5.jar
    3. Go to the etc/apps/SA-ITSI-MetricAD/lib/ directory.
      cd $SPLUNK_HOME/etc/apps/SA-ITSI-MetricAD/lib/
    4. Verify that the checksum of the jar matches the checksum you made note of earlier.
      • ITSI version 4.7.x, 4.9.x, 4.11.0: 
        sha256sum  org.apache.logging.log4j.log4j-core-2.13.2.jar
      • ITSI version 4.4.x: 
        sha256sum  org.apache.logging.log4j.log4j-core-2.3.jar

Workaround steps for self-managed deployments on a search head cluster on Windows

Follow these steps to implement a workaround for CVE-2021-44228 in self-managed deployments of Splunk IT Service Intelligence or IT Essentials Work on search head clusters in a Windows environment. You can use this fix as a workaround until you have time to upgrade to a maintenance version that addresses the vulnerability. These changes require a maintenance window of about eight to ten minutes.

Prerequisite
You must have administrative access to the operating system on the machine where ITSI or IT Essentials Work is installed to perform these steps.

For all search head clusters where ITSI or ITE Work is installed, perform the following procedures.

$SPLUNK_HOME is the directory where you installed Splunk Enterprise.

Steps

  1. Log on to the deployer.
  2. Go to the etc/shcluster/apps/SA-ITOA/lib/java/event_management/libs/ directory. 
    cd $SPLUNK_HOME\etc\shcluster\apps\SA-ITOA\lib\java\event_management\libs\
  3. Run the command that corresponds to your version of ITSI or IT Essentials Work to delete the JndiLookup classes from the log4j jar file.
    • ITSI or IT Essentials Work version 4.7.x, 4.9.x, or 4.11.x: 
      $SPLUNK_HOME\etc\shcluster\apps\SA-ITOA\lib\java\event_management\libs>"C:\Program Files\7-Zip\7z.exe" D .\log4j-core-2.13.2.jar org\apache\logging\log4j\core\lookup\JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.13.2 in this directory, you can safely delete these .jar files with lower versions.

    • ITSI version 4.4.x: 
      $SPLUNK_HOME\etc\shcluster\apps\SA-ITOA\lib\java\event_management\libs>"C:\Program Files\7-Zip\7z.exe" D .\log4j-core-2.5.jar org\apache\logging\log4j\core\lookup\JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.5 in this directory, you can safely delete these .jar files with lower versions.

  4. Make a note of the updated jar's checksum.
    • ITSI or IT Essentials Work version 4.7.x, 4.9.x, or 4.11.x: 
      $SPLUNK_HOME\etc\shcluster\apps\SA-ITOA\lib\java\event_management\libs>"C:\Program Files\7-Zip\7z.exe" h .\log4j-core-2.13.2.jar
    • ITSI version 4.4.x: 
      $SPLUNK_HOME\etc\shcluster\apps\SA-ITOA\lib\java\event_management\libs>"C:\Program Files\7-Zip\7z.exe" h .\log4j-core-2.5.jar
  5. Go to the etc/shcluster/apps/SA-ITSI-MetricAD/lib/ directory. 
    cd $SPLUNK_HOME\etc\shcluster\apps\SA-ITSI-MetricAD\lib\
  6. Run the command that corresponds to your version of ITSI or IT Essentials Work to delete JndiLookup classes from the log4j jar file.
    • ITSI version 4.7.x, 4.9.x, 4.11.0: 
      $SPLUNK_HOME\etc\shcluster\apps\SA-ITSI-MetricAD\lib>"C:\Program Files\7-Zip\7z.exe" D .\org.apache.logging.log4j.log4j-core-2.13.2.jar org\apache\logging\log4j\core\lookup\JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.13.2 in this directory, you can safely delete these .jar files with lower versions.

    • ITSI version 4.4.x: 
      $SPLUNK_HOME\etc\shcluster\apps\SA-ITSI-MetricAD\lib>"C:\Program Files\7-Zip\7z.exe" D .\org.apache.logging.log4j.log4j-core-2.3.jar org\apache\logging\log4j\core\lookup\JndiLookup.class

      Once the above command executes successfully, if there are log4j .jar files with versions lower than 2.3 in this directory, you can safely delete these .jar files with lower versions.

  7. Make note of the updated jar's checksum
    • ITSI version 4.7.x, 4.9.x, 4.11.0: 
      $SPLUNK_HOME\etc\shcluster\apps\SA-ITSI-MetricAD\lib\>"C:\Program Files\7-Zip\7z.exe" h .\org.apache.logging.log4j.log4j-core-2.13.2.jar
    • ITSI version 4.4.x: 
      $SPLUNK_HOME\etc\shcluster\apps\SA-ITSI-MetricAD\lib\>"C:\Program Files\7-Zip\7z.exe" h .\org.apache.logging.log4j.log4j-core-2.3.jar
  8. Delete the older versions of log4j .jar files from the deployer before deploying the updated jars. To view a list of the files and their locations, see Version-specific upgrade notes for ITSI.
  9. Deploy updated jars to search heads with the following command. The -target parameter specifies the URI and management port for any member of the cluster. The -auth parameter specifies credentials for the deployer instance. 
    $SPLUNK_HOME\bin\splunk apply shcluster-bundle -target <URI>:<management_port> -auth <username>:<password>
  10. Wait for deployment and rolling restart to complete.
  11. Log on to the search heads and verify the updated jar's checksums.
    1. Go to the etc/apps/SA-ITOA/lib/java/event_management/libs/ directory. 
      cd $SPLUNK_HOME\etc\apps\SA-ITOA\lib\java\event_management\libs\
    2. Verify that the checksum of the jar matches the checksum you made note of earlier.
      • ITSI or IT Essentials Work version 4.7.x, 4.9.x, or 4.11.x: 
        $SPLUNK_HOME\etc\apps\SA-ITOA\lib\java\event_management\libs>"C:\Program Files\7-Zip\7z.exe" h .\log4j-core-2.13.2.jar
      • ITSI version 4.4.x: 
        $SPLUNK_HOME\etc\apps\SA-ITOA\lib\java\event_management\libs>"C:\Program Files\7-Zip\7z.exe" h .\log4j-core-2.5.jar
    3. Go to the etc/apps/SA-ITSI-MetricAD/lib/ directory. 
      cd $SPLUNK_HOME\etc\apps\SA-ITSI-MetricAD\lib\
    4. Verify that the checksum of the jar matches the checksum you made note of earlier.
      • ITSI version 4.7.x, 4.9.x, 4.11.0: 
        $SPLUNK_HOME\etc\apps\SA-ITSI-MetricAD\lib\>"C:\Program Files\7-Zip\7z.exe" h .\org.apache.logging.log4j.log4j-core-2.13.2.jar
      • ITSI version 4.4.x: 
        $SPLUNK_HOME\etc\apps\SA-ITSI-MetricAD\lib\>"C:\Program Files\7-Zip\7z.exe" h .\org.apache.logging.log4j.log4j-core-2.3.jar
Last modified on 09 March, 2022
Before you upgrade IT Service Intelligence   Upgrade IT Service Intelligence on a single instance

This documentation applies to the following versions of Splunk® IT Service Intelligence: 4.11.0, 4.11.1, 4.12.0 Cloud only, 4.12.1 Cloud only, 4.12.2 Cloud only, 4.13.0, 4.13.1, 4.13.2, 4.13.3, 4.14.0 Cloud only, 4.14.1 Cloud only, 4.14.2 Cloud only, 4.15.0, 4.15.1, 4.15.2, 4.15.3, 4.16.0 Cloud only

Acrobat logo Download manual
Acrobat logo Download this page

Was this topic useful?







You must be logged into splunk.com in order to post comments. Log in now.

Please try to keep this discussion focused on the content covered in this documentation topic. If you have a more general question about Splunk functionality or are experiencing a difficulty with Splunk, consider posting a question to Splunkbase Answers.

0 out of 1000 Characters