Advanced container customization

The Splunk App for Data Science and Deep Learning relies on container images for CPU or GPU based machine learning workloads. The Splunk App for Data Science and Deep Learning ships with certain default images including Golden CPU, and Golden GPU.

Your organization might need to modify, secure, or extend these images for specialized libraries, security requirements, or offline (air-gapped) environments. The splunk-mltk-container-docker GitHub repository provides the Dockerfiles, requirements files, and scripts required to build, customize, and manage these container images.

You can use the repository to build and tailor container images for HPC-level model training, specialized Python dependencies, or compliance with internal security standards. By understanding the scripts, tag_mapping.csv, and Dockerfiles, you can complete the following tasks:

• Add new libraries or pinned dependencies.
• Switch to GPU runtimes or different base OS images.
• Scan containers for vulnerabilities.
• Automate builds with a CI pipeline.
• Integrate seamlessly with the images.conf file for a frictionless DSDL experience.

If you require additional customization such as Red Hat environment compliance, or packaging proprietary libraries, the flexible scripting approach in build.sh, bulk_build.sh, and compile_image_python_requirements.sh ensures you can keep your images in sync with your enterprise policies. Combine this customization with other DSDL advanced workflows to fully operationalize machine learning at scale within the Splunk platform.

About the GitHub repository

The splunk-mltk-container-docker repository hosts the Dockerfiles, scripts, and requirements needed to create the container images used by DSDL.

Review the following table for descriptions of the repository components:

Component	Description
/dockerfiles/	Contains multiple Dockerfiles. Includes files different base operating systems such as Debian and UBI, and library sets such as golden-cpu, minimal-cpu, spark, and rapids. Examples: Dockerfile.debian.golden-cpu installs an array of data science libraries on Debian. Dockerfile.redhat.ubi.golden-cpu does the same for Red Hat UBI9.
/requirements_files/	Houses Python requirements files that define which libraries get installed in each image variant. Typically split into base_requirements and specific_requirements to handle minimal and golden additions.
/scripts/	A suite of shell scripts for building, bulk-building, scanning, and testing images. Examples: The build.sh shell script builds a single image based on a chosen tag from tag_mapping.csv. The bulk_build.sh builds multiple images from a list of tags. The scan_container.sh uses Trivy to scan for vulnerabilities. The test_container.sh can run Playwright-based UI tests on a container.
tag_mapping.csv	A critical .CSV file enumerating build configurations. For example, golden-cpu, golden-gpu, and minimal-cpu. Each row maps a "tag" to the base image, Dockerfile, requirements files, and runtime flags.
/images_conf_files/	Stores the generated images.conf stanzas that can be merged into the Splunk platform /mltk-container/local/ directory. Each .conf file corresponds to a built image, letting DSDL know which images are available.

Scripts overview

Review the following descriptions of the main scripts and how they fit into a typical build pipeline.

All scripts are in the /scripts/ directory of the repository.

build.sh

Overview: Build a single container image using configuration from tag_mapping.csv.
Syntax: ./build.sh <build_configuration_tag> <repo_name> <version>
Parameters:

• build_configuration_tag: References a row in tag_mapping.csv.
• repo_name: Optional prefix for the Docker repo.
• version: A version tag for the final image.
  

Example:./build.sh golden-cpu splunk/ 5.1.1
Workflow:

1. Reads the row for golden-cpu in tag_mapping.csv to find the base image, Dockerfile, and requirements files.
2. Optionally compiles Python requirements if not pre-compiled.
3. Executes docker build with the chosen Dockerfile and context.
4. Generates a .conf file in /images_conf_files/ describing the new image.

bulk_build.sh

Overview: Builds all containers or a subset from a CSV listing.
Key step: Iterates over each line in tag_mapping.csv to call build.sh for each configured tag.
Syntax: ./bulk_build.sh <tag_mapping.csv> <repo_name> <version>
Example: ./bulk_build.sh tag_mapping.csv splunk/ 5.1.1

compile_image_python_requirements.sh

Overview: Pre-compile or "lock" Python dependencies for a given image variant to reduce build-time or avoid dynamic dependency resolution.
Syntax: ./compile_image_python_requirements.sh <tag_name>
Workflow:

1. Builds a minimal environment Dockerfile to resolve Python packages.
2. Outputs a pinned/locked requirements file in requirements_files/compiled_*.txt.
3. Speeds up future builds by installing pinned versions of each library.

test_container.sh

Overview: Run a set of integration tests against a built container using Playwright or other testing frameworks. Can be used to simulate Splunk and Jupyter interactions or to validate container endpoints are running as expected.
Prerequisite: A local Python virtual environment or system Python with the correct dependencies to run Playwright.
Syntax: ./test_container.sh <tag_name> <repo_name> <version>

scan_container.sh

Overview: Use Trivy to scan the built container for vulnerabilities.
Syntax: ./scan_container.sh <tag_name> <repo_name> <version>
Benefits:

• Identifies potential CVEs or insecure packages in the final image.
• Helpful for production-grade images to ensure compliance with security standards.

Tag mapping and CSV configuration

The tag_mapping.csv orchestrates the following build logic:

Column name	Details
Tag	Short name for the image variant. For example golden-cpu, minimal-cpu, or ubi-golden-cpu. Used as <build_configuration_tag> in build.sh.
base_image	Base operating system (OS) image. For example debian:bullseye or registry.access.redhat.com/ubi9/ubi:latest. Must be accessible to docker pull.
dockerfile	The Dockerfile to use. For example Dockerfile.debian.golden-cpu. Located in /dockerfiles/.
base_requirements	The base Python requirements file. For example base_functional.txt. Found in /requirements_files/.
specific_requirements	Additional specialized libraries. For example specific_golden_cpu.txt. Usually large ML libs like TensorFlow or PyTorch.
runtime	Is none or nvidia for GPU usage. If nvidia the script sets up GPU libraries.
requirements_dockerfile	Optional Docker file used for pre-compiling Python dependencies. For example Dockerfile.debian.requirements.

The following is an example of the content found within 1 row of the tag_mapping.csv file:

Tag,base_image,dockerfile,base_requirements,specific_requirements,runtime,requirements_dockerfile
golden-cpu,deb:bullseye,Dockerfile.debian.golden-cpu,base_functional.txt,specific_golden_cpu.txt,none,Dockerfile.debian.requirements

Customizing images

Review the options for customizing container images.

Adding extra libraries

If you need a library such as pyarrow or transformers that is not included in specific_golden_cpu.txt you can complete these steps:

1. Fork or clone the repository.
2. Edit or create a new requirements_files/ text file with your library pinned, for example:

   # my_custom_libraries.txt
   pyarrow==10.0.1
   transformers==4.25.1
   

3. Create or edit a row in tag_mapping.csv referencing your new file, for example:

   Tag,base_image,dockerfile,base_requirements,specific_requirements,runtime,requirements_dockerfile
   golden-cpu-custom,deb:bullseye,Dockerfile.debian.golden-cpu,base_functional.txt,my_custom_libraries.txt,none,Dockerfile.debian.requirements

4. Build the custom variant, for example:

   ./build.sh golden-cpu-custom splunk/ 5.2.0

5. After building, your new image is tagged as splunk/golden-cpu-custom:5.2.0, plus a .conf file is created in images_conf_files/ that you can merge into the Splunk platform images.conf.

Red Hat Universal Base Image (UBI) minimal approach

If you need a Red Hat UBI9 base for enterprise compliance, complete these steps:

1. Select a row or create a row referencing Dockerfile.redhat.ubi.golden-cpu.
2. Edit the ubi-minimal or ubi-golden-cpu Dockerfile for your internal repos and set tag_mapping.csv accordingly.

Air-gapped deployments

If your deployment is air-gapped or offline, complete these steps:

1. Bulk build images on an internet-connected machine using bulk_build.sh.
2. Scan the images with scan_container.sh.
3. Push to a local registry or save as .TAR using Docker Save.
4. Transfer to the offline environment and use Docker Load.
5. Update the Splunk platform images.conf to point to your internal registry references.

Container customization guidelines

Consider the following guidelines when using customized containers:

Component	Guideline
Python dependency conflicts	Some advanced ML libraries conflict with older ones. Always run scan_container.sh and consider using compile_image_python_requirements.sh to lock consistent versions.
Large image sizes	Golden CPU/GPU images can be multiple GB in size. Consider minimal images if you only need a subset of libraries.
Requirements Dockerfile	If you update requirements_files/, remove or regenerate compiled files in /requirements_files/compiled_* or they won't reflect new pins.
No official support	Some scripts and Dockerfiles are unofficial or community features. The Splunk platform only fully supports the official DSDL containers for standard usage. Your custom builds are used at your own risk.
Security hardening	For production, consider scanning your images frequently and applying OS-level hardening. The scan_container.sh script is useful, but you can also consider removing unneeded packages or reduce root privileges in Dockerfiles.
Version management	Maintain a separate branch or fork of splunk-mltk-container-docker. Tag each commit with the container version you produce so you can replicate or revert builds if needed.

Example: Container customization

The following is an example workflow for a custom golden-cpu image with pinned requirements:

1. Clone the repo:

   git clone https://github.com/splunk/splunk-mltk-container-docker
   cd splunk-mltk-container-docker
   

2. Create or edit your row in tag_mapping.csv:

   Tag,base_image,dockerfile,base_requirements,specific_requirements,runtime,requirements_dockerfile
   golden-cpu-custom,deb:bullseye,Dockerfile.debian.golden-cpu,base_functional.txt,my_custom_libraries.txt,none,Dockerfile.debian.requirements
   

3. (Optional) Pre-compile Python requirements:

   ./compile_image_python_requirements.sh golden-cpu-custom
   

4. Build the new image:

   ./build.sh golden-cpu-custom splunk/ 5.2.0
   

5. Scan the built image:

   ./scan_container.sh golden-cpu-custom splunk/ 5.2.0
   

6. Push the image to your Docker registry:

   docker push splunk/golden-cpu-custom:5.2.0
   

7. Copy the generated .conf snippet in ./images_conf_files/ into your Splunk search head mltk-container/local/images.conf file.
8. Restart Splunk or reload DSDL to see the new container listed.
9. Use the new container in DSDL commands:

   index=my_data
   | fit MLTKContainer algo=barebone_template mode=stage into app:MyNewModel container_image="splunk/golden-cpu-custom:5.2.0"