Set up the Splunk App for Data Science and Deep Learning using AWS and EKS

Amazon Web Services (AWS) Elastic Kubernetes Service (EKS) is a frequently used managed service for running container workloads in the Cloud. Splunk App for Data Science and Deep Learning (DSDL) customers can use this managed service on AWS, connect DSDL to EKS, and run workloads there.

Prerequisites

The following is required for a successful set up:

• An AWS account with sufficient administrator permissions
• DSDL app installed and configured on your Splunk platform instance

Set up steps

Complete these steps to use DSDL with AWS:

• Create a new IAM user and role
• Install and configure the AWS CLI
• Install the eksctl command line utility
• Create an Amazon EKS cluster
• Set up an EFS CSI driver in EKS
• Connect DSDL to the AWS EKS cluster
• (Optional) Configure Observability

Create a new IAM user and role

AWS Identity and Access Management (IAM) is an Amazon web service that helps you securely control access to any AWS resources. IAM enables shared access to your AWS account and resources while protecting your AWS account root-user information.

1. In your AWS account, create a new user. Create a user name that identifies it as DSDL related, for example, dltk-admin.
2. For the AWS Credential type, select Access key - programmatic access.
3. For permissions, choose Add user to group and Create group. For the permission boundary, choose "Create user without a permission boundary".
4. Create a name for the new group that identifies it as DSDL related, for example, dltk-admin-group. Select "AdministratorAccess" from the policy menu.
5. Click Create group.
6. Add the user you created in step 1 to the group you created in step 3. When complete, keep the confirmation page open in order to use the Access Key ID and Secret Access Key in the next section.

Install and configure the AWS CLI

Download and configure the AWS command line interface using the Access Key ID and Secret Access Key login credentials from the previous step. For more information, see https://aws.amazon.com/cli/

Install the eksctl command line utility

You can use Amazon Elastic Kubernetes Service (Amazon EKS) to run Kubernetes on AWS without installing or maintaining your own Kubernetes control plane or nodes. The eksctl command line tool is used for creating and managing Kubernetes clusters on Amazon EKS.

For more information,, see https://docs.aws.amazon.com/eks/latest/userguide/eksctl.html and https://eksctl.io/introduction/#installation

Create an Amazon EKS cluster

In Amazon EKS, create a cluster. You can refer to the following example for guidance:

eksctl create cluster --name dltk-cluster --version 1.21 --region eu-central-1 --nodegroup-name dltk-nodes --node-type t2.xlarge --nodes 2

Add the cluster to Amazon EKS and verify the cluster is up and running.

Set up an EFS CSI driver in EKS

Verify you can use the Storage Class of efs-sc and set up and configure the Amazon Elastic File System (EFS) Container Storage Interface (CSI) driver.

For more information, see https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html

Connect DSDL to the AWS EKS cluster

Complete the following steps:

1. In DSDL, go to Configuration > Setup and scroll down to the input panel for Kubernetes. Fill in the fields to connect DSDL to the AWS EKS cluster, using the cluster name created in a previous step.
   

   On Splunk Cloud the inbound and outbound traffic to reach the cluster and its endpoints must be enabled. For example, using ACS. For more information, see Splunk Cloud Self-Service: Announcing The New Admin Config Service API.

2. From Configuration > Containers , launch a development (DEV) container and verify the Kubernetes container is running.
   




3. In AWS EKS, verify that the DEV deployment exists.
   




4. If you are setting up DSDL for the first time, verify that one of the Examples is running without any errors. Running an example after the container environment setup is a simple way to confirm that an algorithm can run.
   

(Optional) Configure Observability

If you want to use Splunk Observability Cloud (O11y) to monitor the Kubernetes that DSDL uses take steps to configure observability. To learn more about this product offering, see Splunk Observability.

Complete the following steps:

1. In Splunk Observability Cloud configure a new integration as shown in the following image.
   Complete the Configure Integration fields as follows:

   Field name	Value
   Splunk Access Token	Default
   Cluster Name	Enter the name of the cluster you created in an earlier step
   Provider	Amazon Web Services
   Distribution	Amazon EKS
   Add Gateway	No
   Log Collection	True

2. Next, follow the Install Integration steps as shown in the following image.
3. Lastly, look over the Review Inventory page for the cluster added as shown in the following image. Pay attention to CPU utilization on the cluster when running a DSDL container workload.