Set up LLM-RAG in an air-gapped environment

If you use an air-gapped environment, follow these steps to set up LLM-RAG :

• Download required packages
• Complete the set up

Download required packages

On an instance connected to the internet, download the following packages and copy them to the air-gapped instance:

• Docker Service and Docker Compose plugin
• Milvus container images
• Ollama container image
• DSDL LLM-RAG container image
• Sentence-transformers embedding model

Docker Service and Docker Compose plugin

Check the Docker download page based on your instance type. For example, https://download.docker.com/linux/ubuntu/dists/noble/pool/stable/amd64/

Perform the following steps:

1. Run the following command:

   wget 
   https://download.docker.com/linux/ubuntu/dists/noble/pool/stable/amd64/containerd.io_1.6.28-2_amd64.deb 
   wget 
   https://download.docker.com/linux/ubuntu/dists/noble/pool/stable/amd64/docker-buildx-plugin_0.13.1-1~ubuntu.24.04~noble_amd64.deb
   wget 
   https://download.docker.com/linux/ubuntu/dists/noble/pool/stable/amd64/docker-ce-cli_26.0.0-1~ubuntu.24.04~noble_amd64.deb   
   wget 
   https://download.docker.com/linux/ubuntu/dists/noble/pool/stable/amd64/docker-ce_26.0.0-1~ubuntu.24.04~noble_amd64.deb 
   wget 
   https://download.docker.com/linux/ubuntu/dists/noble/pool/stable/amd64/docker-compose-plugin_2.25.0-1~ubuntu.24.04~noble_amd64.deb

2. Copy the *.deb files to your air-gapped instance.

Milvus container images

Perform the following steps:

1. Download the Milvus container images as well as the Docker Compose file for initializing the containers:

   docker pull quay.io/coreos/etcd:v3.5.5
   docker pull minio/minio:RELEASE.2023-03-20T20-16-18Z
   docker pull milvusdb/milvus:v2.4.1
   docker save --output etcd.tar quay.io/coreos/etcd:v3.5.5
   docker save --output minio.tar minio/minio:RELEASE.2023-03-20T20-16-18Z
   docker save --output milvus.tar milvusdb/milvus:v2.4.1
   wget
   https://raw.githubusercontent.com/splunk/splunk-mltk-container-docker/v5.2/beta_content/passive_deployment_prototypes/prototype_ollama_example/compose_files/milvus-docker-compose.yml
   

2. Copy the *.tar files as well as the milvus-docker-compose.yml file to your air-gapped instance.

Ollama container image

Perform the following steps:

1. Download the Ollama container image as well as Docker Compose files:

   docker pull ollama/ollama:latest
   docker save --output ollama.tar ollama/ollama:latest
   wget https://raw.githubusercontent.com/huaibo-zhao-splunk/splunk-mltk-container-docker/huaibo-ollama/beta_content/passive_deployment_prototypes/prototype_ollama_example/compose_files/ollama-docker-compose.yml
   wget 
   https://raw.githubusercontent.com/splunk/splunk-mltk-container-docker/v5.2/beta_content/passive_deployment_prototypes/prototype_ollama_example/compose_files/milvus-docker-compose.yml
   

2. Copy the ollama.tar file as well as the *.yml files to your air-gapped instance.

DSDL LLM-RAG container image

Perform the following steps:

1. Download the LLM-RAG container image:

   docker pull splunk/mltk-container-ubi-llm-rag:5.2.0
   docker save --output mltk-container-ubi-llm-rag.tar 
   splunk/mltk-container-ubi-llm-rag:5.2.0
   

2. Copy the mltk-container-ubi-llm-rag.tar file to your air-gapped instance.

Sentence-transformers embedding model

Perform the following steps:

1. Navigate to the huggingface page of the model you want to use, download all the related files, and package up the files. For example, go to https://huggingface.co/intfloat/multilingual-e5-large/tree/main and check all the files to download.

   mkdir -p intfloat/multilingual-e5-large
   cd intfloat/multilingual-e5-large
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/config.json
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/model.safetensors
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/modules.json
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/pytorch_model.bin
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/sentence_bert_config.json
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/sentencepiece.bpe.model
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/special_tokens_map.json
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/tokenizer.json
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/tokenizer_config.json
   mkdir 1_Pooling
   cd 1_Pooling/
   wget https://huggingface.co/intfloat/multilingual-e5-large/resolve/main/1_Pooling/config.json
   cd ../../..
   tar -cvf model_files.tar intfloat
   

2. Copy the model_files.tar file to your air-gapped instance.

Complete the set up

Perform the following steps:

1. Extract from the TAR archive (untar) the Milvus images and run Docker Compose:

   docker load --input etcd.tar
   docker load --input minio.tar
   docker load --input milvus.tar
   docker compose -f milvus-docker-compose.yml up --detach
   

2. Untar the Ollama image and run Docker Compose:

   docker load --input ollama.tar
   (CPU): docker compose -f ollama-docker-compose.yml up --detach
   (GPU): docker compose -f ollama-docker-compose-gpu.yml up --detach
   

3. Untar the DSDL image:

   docker load --input mltk-container-ubi-llm-rag.tar
   docker run -d --rm \
       -p 8888:8888 \
       -p 6006:6006 \
       -p 6060:6000 \
       -p 4040:4040 \
       -p 5000:5000 \
       --volume mltk-container-data:/srv:rw \
       --volume mltk-container-app:/srv/backup/app:ro \
       --volume mltk-container-notebooks:/srv/backup/notebooks:ro \
       --network dsenv-network \
   splunk/mltk-container-ubi-llm-rag:5.2.0
   

4. Untar the model file at Docker volume:

   tar -xvf model_files.tar -C /var/lib/docker/volumes/mltk-container-data/_data/app/model/data