Update images used in testing an airgapped CKF 1.9 deployment (#1053)

* Update images for testing in airgapped

* Update tf-mnist-with-summaries image used for testing in airgapped

* Update file structure to support aigapped tests for both CKF 1.8 and 1.9

* Add new image for pipelines

* Update tests/airgapped/README.md

* Update tests/airgapped/1.9/pipelines/pipelines-runner/Dockerfile

Co-authored-by: Noha Ihab <[email protected]>

* Update kfp SDK version for pipelines runner

* Update tests/airgapped/1.9/training/README.md

Co-authored-by: Noha Ihab <[email protected]>

* Update READMEs for airgapped testing

* Make tests/airgap/setup/setup.sh non-interactive

* Update default Juju version

* Make tests/airgap/setup/setup.sh non-interactive

* Remove unneeded environment variable on setup.sh

* Update READMEs

* Update READMEs in the scripts directory

* Update README in scripts/airgapped

* Update hyperlinks to get-all-images.py

* Update README for the pipelines-runner image

* Update for pipelines

* Update airgap.sh to properly receive --testing-images-path option

* Change default name for the testing images files

* Update README

* Update setup script to add dependencies

* Remove dependencies from setup.sh

* Don't exit if docker group already exists

* Remove debugging echo statements

* Remove README of knative

---------

Co-authored-by: Noha Ihab <[email protected]>

scripts/README.md

@@ -6,11 +6,20 @@ This directory contains helper scripts for Charmed Kubeflow, during CI and not o
 
 You can get a list of all the OCI images used by the bundle by running the following command:
 ```bash
-pip3 install -r scritps/requirements.txt
+pip3 install -r scripts/requirements.txt
 
 python3 scripts/get-all-images.py \
-    --append-images tests/airgapped/ckf-1.8-testing-images.txt \
-    releases/1.8/stable/kubeflow/bundle.yaml \
+    --append-images tests/airgapped/1.9/testing-images.txt \
+    releases/1.9/stable/bundle.yaml \
+    > images-all.txt
+```
+For Charmed Kubeflow 1.8, run
+```bash
+pip3 install -r scripts/requirements.txt
+
+python3 scripts/get-all-images.py \
+    --append-images tests/airgapped/1.8/testing-images.txt \
+    releases/1.9/stable/kubeflow/bundle.yaml \
     > images-all.txt
 ```
 

scripts/airgapped/README.md

@@ -26,15 +26,24 @@ Use the following script to get the list of all OCI images used by a bundle.
 This script makes the following assumptions:
 1. Every charm in the bundle has a `_github_repo_name` metadata field,
    containing the repository name of the charm (the org is assumed to be
-   canonical)
+   canonical).
 2. Every charm in the bundle has a `_github_repo_branch` metadata field,
-   containing the branch of the source code
+   containing the branch of the source code.
 3. There is a script called `tools/get_images.sh` in each repo that gathers
-   the images for that repo
+   the images for that repo.
 
+For Charmed Kubeflow 1.9, run:
 ```bash
 python3 scripts/get-all-images.py \
-    --append-images=tests/airgapped/ckf-1.8-testing-images.txt \
+    --append-images=tests/airgapped/1.9/testing-images.txt \
+    releases/1.9/stable/bundle.yaml \
+    > images.txt
+```
+
+For Charmed Kubeflow 1.8, run:
+```bash
+python3 scripts/get-all-images.py \
+    --append-images=tests/airgapped/1.8/testing-images.txt \
     releases/1.8/stable/kubeflow/bundle.yaml \
     > images.txt
 ```

scripts/airgapped/prerequisites.sh

@@ -13,7 +13,7 @@ sudo apt update
 # See in the tests/airgapped README.md instructions for the docker user changes to take effect.
 echo "Installing Docker"
 sudo snap install docker
-sudo groupadd docker
+sudo groupadd -f docker
 sudo usermod -aG docker $USER
 sudo snap disable docker
 sudo snap enable docker

tests/airgapped/katib/README.md → tests/airgapped/1.8/katib/README.md

@@ -6,7 +6,7 @@ This directory is dedicated to testing Katib in an airgapped environment.
 
 Prepare the airgapped environment and deploy CKF by following the steps in [Airgapped test scripts](https://github.com/canonical/bundle-kubeflow/tree/main/tests/airgapped#testing-airgapped-installation).
 
-Once you run the test scripts, the `kubeflowkatib/simple-pbt:v0.16.0` image used in the `simple-pbt` experiment will be included in your airgapped environment. It's specifically added in the [`get-all-images.py` script](../../../scripts/get-all-images.py).
+Once you run the test scripts, the `kubeflowkatib/simple-pbt:v0.16.0` image used in the `simple-pbt` experiment will be included in your airgapped environment. It's specifically added in the [`get-all-images.py` script](../../../../scripts/get-all-images.py).
 
 ## How to test Katib in an Airgapped environment
 1. Connect to the dashboard by visiting the IP of your airgapped VM. To get the IP run:

tests/airgapped/knative/README.md → tests/airgapped/1.8/knative/README.md

@@ -6,7 +6,7 @@ This directory is dedicated to testing Knative in an airgapped environment.
 
 Prepare the airgapped environment and deploy CKF by following the steps in [Airgapped test scripts](https://github.com/canonical/bundle-kubeflow/tree/main/tests/airgapped#testing-airgapped-installation).
 
-Once you run the test scripts, the `knative/helloworld-go` image used in the `helloworld` example will be included in your airgapped environment. It's specifically added in the [`get-all-images.py` script](../../../scripts/get-all-images.py).
+Once you run the test scripts, the `knative/helloworld-go` image used in the `helloworld` example will be included in your airgapped environment. It's specifically added in the [`get-all-images.py` script](../../../../scripts/get-all-images.py).
 
 ## How to test Knative in an Airgapped environment
 1. Connect to the dashboard by visiting the IP of your airgapped VM. To get the IP run:

tests/airgapped/1.8/pipelines/README.md

@@ -0,0 +1,17 @@
+# Testing Pipelines in Airgapped
+
+## The `kfp-airgapped-ipynb` Notebook
+To test Pipelines in Airgapped, we are using the Notebook in this directory. It contains the Data passing pipeline example, with the configuration of the Pipeline components to use the `pipelines-runner` [image](./pipelines-runner/README.md).
+
+The `pipelines-runner` image will be included in your airgapped environment given that you used the [Airgapped test scripts](../../README.md). It's specifically added in the [`get-all-images.py` script](../../../../scripts/airgapped/get-all-images.py).
+
+## How to test Pipelines in an Airgapped environment
+1. Prepare the airgapped environment and Deploy CKF by following the steps in [Airgapped test scripts](../../README.md).
+2. Connect to the dashboard by visiting the IP of your airgapped VM. To get the IP run:
+```
+lxc ls | grep eth0
+```
+3. Go to the `Notebooks` tab, create a new Notebook server and choose `jupyter-tensorflow-full` image from the dropdown.
+4. Connect to the Notebook server and upload the `kfp-airgapped-ipynb` Notebook.
+5. Run the Notebook.
+6. Click on `Run details` from the output of the last cell in the Notebook to see the status of the Pipeline run.

tests/airgapped/pipelines/kfp-airgapped.ipynb → tests/airgapped/1.8/pipelines/kfp-airgapped.ipynb

@@ -49,7 +49,7 @@
    },
    "outputs": [],
    "source": [
-    "@component(base_image='172.17.0.2:5000/pipelines-runner:ckf-1.8')\n",
+    "@component(base_image='172.17.0.2:5000/charmedkubeflow/pipelines-runner:ckf-1.8')\n",
     "def preprocess(\n",
     "    # An input parameter of type string.\n",
     "    message: str,\n",
@@ -104,7 +104,7 @@
    },
    "outputs": [],
    "source": [
-    "@component(base_image='172.17.0.2:5000/pipelines-runner:ckf-1.8')\n",
+    "@component(base_image='172.17.0.2:5000/charmedkubeflow/pipelines-runner:ckf-1.8')\n",
     "def train(\n",
     "    # Use InputPath to get a locally accessible path for the input artifact\n",
     "    # of type `Dataset`.\n",

tests/airgapped/training/README.md → tests/airgapped/1.8/training/README.md

@@ -4,9 +4,9 @@ This directory is dedicated to testing training operator in an airgapped environ
 
 ## Prerequisites
 
-Prepare the airgapped environment and deploy CKF by following the steps in [Airgapped test scripts](../README.md#testing-airgapped-installation).
+Prepare the airgapped environment and deploy CKF by following the steps in [Airgapped test scripts](../../README.md#testing-airgapped-installation).
 
-Once you run the test scripts, the `kubeflow-ci/tf-mnist-with-summaries:1.0` image used in the `tfjob-simple` training job will be included in your airgapped environment. It's specifically added in the [`get-all-images.py` script](../../../scripts/get-all-images.py).
+Once you run the test scripts, the `kubeflow-ci/tf-mnist-with-summaries:1.0` image used in the `tfjob-simple` training job will be included in your airgapped environment. It's specifically added in the [`get-all-images.py` script](../../../../scripts/get-all-images.py).
 
 ## How to test training operator in an Airgapped environment
 1. Connect to the dashboard by visiting the IP of your airgapped VM. To get the IP run:
@@ -18,11 +18,11 @@ Once you run the test scripts, the `kubeflow-ci/tf-mnist-with-summaries:1.0` ima
 2. Log in to the dashboard and create a Profile.
 3. Apply the `tfjob-simple.yaml` found in this directory to your Profile's Namespace:
 ```
-microk8s.kubectl apply -f ./tfjob-simple.yaml -n <your namespace>
+microk8s kubectl apply -f ./tfjob-simple.yaml -n <your namespace>
 ```
 4. Wait for the tfjob to be `Succeeded`
 ```
-microk8s.kubectl get tfjob -n <your namespace>
+microk8s kubectl get tfjob -n <your namespace>
 ```
 Expected output:
 ```

tests/airgapped/1.9/katib/README.md

@@ -0,0 +1,22 @@
+# Testing Katib in airgapped
+
+This directory is dedicated to testing Katib in an airgapped environment.
+
+## Prerequisites
+
+Prepare the airgapped environment and deploy CKF by following the steps in [Airgapped test scripts](https://github.com/canonical/bundle-kubeflow/tree/main/tests/airgapped#testing-airgapped-installation).
+
+Once you run the test scripts, the `kubeflowkatib/simple-pbt:v0.17.0` image used in the `simple-pbt` experiment will be included in your airgapped environment. It's specifically added in the [`get-all-images.py` script](../../../../scripts/airgapped/get-all-images.py).
+
+## How to test Katib in an Airgapped environment
+1. Connect to the dashboard by visiting the IP of your airgapped VM. To get the IP run:
+    ```
+    lxc ls | grep eth0
+    ```
+    Look for the IP of the `airgapped-microk8s` instance.
+
+2. Log in to the dashboard and create a Profile.
+3. Go to the `Katib Experiments` tab from the dashboard sidebar.
+4. Click `New Experiment` then `Edit and submit YAML`.
+5. Paste the contents of the `simple-pbt.yaml` file found in this directory.
+6. Create the Experiment, and monitor its status to check it is `Succeeded`.

tests/airgapped/1.9/katib/simple-pbt.yaml

@@ -0,0 +1,64 @@
+# Source: katib/examples/v1beta1/hp-tuning/simple-pbt.yaml
+# This example is slightly modified from upstream to consume less resources
+# and disable istio sidecar.
+# There's a `modified` comment where we diverge from upstream.
+# When updating this file, make sure to keep those modifications.
+---
+apiVersion: kubeflow.org/v1beta1
+kind: Experiment
+metadata:
+  name: simple-pbt
+spec:
+  maxTrialCount: 1  # modified
+  parallelTrialCount: 1  # modified
+  maxFailedTrialCount: 1  # modified
+  resumePolicy: FromVolume
+  objective:
+    type: maximize
+    goal: 0.99
+    objectiveMetricName: Validation-accuracy
+  algorithm:
+    algorithmName: pbt
+    algorithmSettings:
+      - name: suggestion_trial_dir
+        value: /var/log/katib/checkpoints/
+      - name: n_population
+        value: '40'
+      - name: truncation_threshold
+        value: '0.2'
+  parameters:
+    - name: lr
+      parameterType: double
+      feasibleSpace:
+        min: '0.0001'
+        max: '0.02'
+        step: '0.0001'
+  trialTemplate:
+    primaryContainerName: training-container
+    trialParameters:
+      - name: learningRate
+        description: Learning rate for training the model
+        reference: lr
+    trialSpec:
+      apiVersion: batch/v1
+      kind: Job
+      spec:
+        template:
+          spec:
+            containers:
+              - name: training-container
+                image: 172.17.0.2:5000/kubeflowkatib/simple-pbt:v0.17.0
+                command:
+                  - "python3"
+                  - "/opt/pbt/pbt_test.py"
+                  - "--epochs=20"
+                  - "--lr=${trialParameters.learningRate}"
+                  - "--checkpoint=/var/log/katib/checkpoints/"
+                resources:  # modified
+                  limits:  # modified
+                    memory: "2Gi"  # modified
+                    cpu: "1"  # modified
+            restartPolicy: Never
+          metadata: # modified
+            annotations:  # modified
+              sidecar.istio.io/inject: "false"  # modified

tests/airgapped/1.9/knative/README.md

@@ -0,0 +1,38 @@
+# Testing Knative in airgapped
+
+This directory is dedicated to testing Knative in an airgapped environment.
+
+## Prerequisites
+
+Prepare the airgapped environment and deploy CKF by following the steps in [Airgapped test scripts](https://github.com/canonical/bundle-kubeflow/tree/main/tests/airgapped#testing-airgapped-installation).
+
+Once you run the test scripts, the `knative/helloworld-go` image used in the `helloworld` example will be included in your airgapped environment. It's specifically added in the [`get-all-images.py` script](../../../../scripts/get-all-images.py).
+
+## How to test Knative in an Airgapped environment
+1. Connect to the dashboard by visiting the IP of your airgapped VM. To get the IP run:
+```
+lxc ls | grep eth0
+```
+2. Log in to the dashboard and create a Profile.
+3. Apply the `helloworld.yaml` found in this directory to your Profile's Namespace:
+```
+microk8s kubectl apply -f ./helloworld.yaml -n <your namespace>
+```
+4. Wait for the Knative Service to be `Ready`
+```
+microk8s kubectl get ksvc -n <your namespace>
+```
+Expected output:
+```
+NAME    URL                                      LATESTCREATED   LATESTREADY   READY   REASON
+hello   http://hello.admin.10.64.140.43.nip.io   hello-00001     hello-00001   True
+```
+
+5. Curl the Knative Service using the `URL` from the previous step
+```
+curl -L http://hello.admin.10.64.140.43.nip.io
+```
+Expected output:
+```
+Hello World!
+```

tests/airgapped/1.9/knative/helloworld.yaml

@@ -0,0 +1,19 @@
+# This example is based on the [Knative Serving tutorial](https://knative.dev/docs/getting-started/first-service/#__tabbed_1_2)
+apiVersion: serving.knative.dev/v1
+kind: Service
+metadata:
+  name: hello
+spec:
+  template:
+    metadata:
+      labels:
+        # Disable istio sidecar due to https://github.com/canonical/kserve-operators/issues/216
+        sidecar.istio.io/inject	: "false"
+    spec:
+      containers:
+        - image: 172.17.0.2:5000/knative/helloworld-go:latest
+          ports:
+            - containerPort: 8080
+          env:
+            - name: TARGET
+              value: "World"

tests/airgapped/pipelines/README.md → tests/airgapped/1.9/pipelines/README.md

@@ -3,15 +3,15 @@
 ## The `kfp-airgapped-ipynb` Notebook
 To test Pipelines in Airgapped, we are using the Notebook in this directory. It contains the Data passing pipeline example, with the configuration of the Pipeline components to use the `pipelines-runner` [image](./pipelines-runner/README.md).
 
-The `pipelines-runner` image will be included in your airgapped environment given that you used the [Airgapped test scripts](../README.md). It's specifically added in the [`get-all-images.py` script](../../../scripts/get-all-images.py).
+The `pipelines-runner` image will be included in your airgapped environment given that you used the [Airgapped test scripts](../README.md). It's specifically added in the [`get-all-images.py` script](../../../../scripts/get-all-images.py).
 
 ## How to test Pipelines in an Airgapped environment
-1. Prepare the airgapped environment and Deploy CKF by following the steps in [Airgapped test scripts](../README.md).
+1. Prepare the airgapped environment and Deploy CKF by following the steps in [Airgapped test scripts](../../README.md).
 2. Connect to the dashboard by visiting the IP of your airgapped VM. To get the IP run:
 ```
 lxc ls | grep eth0
 ```
-3. Create a Notebook server and choose `jupyter-tensorflow-full` image from the dropdown
-4. Connect to the Notebook server and upload the `kfp-airgapped-ipynb` Notebook
-5. Run the Notebook
+3. Go to the `Notebooks` tab, create a new Notebook server and choose `jupyter-tensorflow-full` image from the dropdown.
+4. Connect to the Notebook server and upload the `kfp-airgapped-ipynb` Notebook.
+5. Run the Notebook.
 6. Click on `Run details` from the output of the last cell in the Notebook to see the status of the Pipeline run.