Kubernetes Operator for Armada and Helm

Introduction

This README is mainly used a log / wiki of results and challenges encounted during the POC

operator-sdk vs kubebuilder.

Things to clean up: Did not had time to sort how to completly migrate from operator-sdh to kubebuilder. Most of the scaffolding is done using the operator-sdk but once the default files are created, the build process mainly relies on kubebuilder

armada-operator code directory structure

cmd

Contains the main.go for the armada operator

pkg/apis/

Contains the golang definition of the CRDs. make generate will recreate the yaml definitions of the CRDs that have to be provided to kubectl in order to deploy the new CRDs. This current version of the operator uses "act" for shortname of ArmadaChart, "acg" as shortname for ArmadaChartGroup and "amf" as shortname for ArmadaManifest.

The first version of the golang code has generated using tool such as "schema-generate" from the schema definition provided with airship-armada project.

pkg/services

Contains the bulk of the interfaces used by the armada controller.

pkg/helm, pkg/helmv2 and pkg/helmv3

To get the process doing, some of the code for ArmadaChart handling is coming from the operator-sdk helm-operator. That code is relying on tiller component which is gone for Helm3. Hence the three directory helm (Interface and Common code), helmv2 (tiller) and helmv3.

The golang package structure is different between helmv2 and helmv3. The Armada Operator will most likely ultimatly have support two branches. In order to delay that milestone, the golang code has been instrumentated with "v2" and "v3" tags which allows to compile either the helm v3 version of the operator or the helm v3 version.

pkg/armada directory

Mainly contain the code for ArmadaChartGroup, ArmadaManifest as will ArmadaBackupLocation handling

pkg/controller directory

Contains the controller and the "Reconcile" functions for ArmadaChart, ArmadaChartGroup and ArmadaManifest. There are currently three controllers (act-controller, acg-controller and amf-controller).

Code changes.

Adjusting the ArmadaOperator CRDs

Upon change of the CRD golang definition, the yaml files have to be regenerated

Note 1: Don't understand yet how to build using operator-sdk operator with the same level of detailes than controller-gen. Big hack that have to be included in Makefile.

Note 2: The generation tool seems to comply with some of OpenAPI specs. The "validation" schema added to in the CRD yaml definition does not contain fields using underscore. Most of those fields containing underscore where defined such a way in the original airship-armada.

make generate

Compiling the armada-operator

To keep the directory tree ligthweight, the vendor directory is not checked in in the current repo. TODO: Since the operator is only using one git branch, the developer has to comment out the helmv2 and add the helmv3 in Gopkg.toml if he wants to build the helmv3 version of the operator. This is still WIP.

dep ensure

To build the v2 version

make docker-build-v2

To build the v3 version

make docker-build-v3

Run unit test and preintegration tests.

If you installed kubebuilder on your system, you will have access to a standalone apiserver, etcd server and kubectl.

Because of a lack of time, the current makefile test statement, will attempt to stop your kubelet and associated container in your local kubernetes cluster, before starting apiserver, etcdserver. TODO: We still need to figure out if it necessary

In order to run the unit tests and the e2e integration tests:

make unittest

Deploying the operator.

Note the current deployment of the operator relies itself on helm.

To install the helm v2 version

make install-v2

To install the helm v3 version

make install-v3

Testing the armada-controller

helm-charts/testchart directory

For testing purpose the current Docker file includes a dummy chart deliverd under armada-charts. This removes the needs to access external chart repository which is also an aspect of helm changing from 2 to 3.

examples/armada

In that directory, the ArmadaChart are enabled by default and the charts are installed as soon as the ArmadaChart CRD are created.

Deployment

Upon creation of the custom resource, the controller will

• Deploy the Armada Manifest described in the CRD
• Update status of the custom resources.
• Add events to the custom resources.

kubectl create -f examples/armada
kubectl describe amf/simple-armada
kubectl get amf
kubectl get acg
kubectl get act

Test controller reconcilation logic (for depending resources)

Upon deletion of its depending resources, the controller will recreate it,

kubectl delete deployment.apps/blog-2-testchart
kubectl get all
kubectl describe act blog-2

Test controller reconcilation logic (for CRD)

When deleting the CRD, the corresponding Armada Manifest should be uninstalled.

kubectl delete -f simple/armada

examples/stepbystep

This directory contains invidual act,acg and amf files which allow the "step" by "step" testing of the deployment.

examples/argo

In that directory, the ArmadaChart (act) are disabled by default and the charts not installed automatically when the act CR are created. This example assumes that the argo controller has been installed. When the "argo worflow" CR is created, the "argo controller" is waked up and it orchestrates the enablement of the ArmadaChart according to the worflow.

Note: You need to have argo installed in your cluster.

kubectl apply -f example/argo

armadachart.armada.airshipit.org/blog-1 created
armadachart.armada.airshipit.org/blog-2 created
workflow.argoproj.io/wf-blog-group created

The first ArmadaChart is installed:

kubectl get all

pod/armada-operator-cbbc7d7f7-zxj5n     1/1     Running             0          60s
pod/blog-1-testchart-5dd8c474f4-26574   0/1     ContainerCreating   0          6s
pod/wf-blog-group-1193326311            0/1     Completed           0          8s
pod/wf-blog-group-2026876860            0/1     ContainerCreating   0          1s
pod/wf-blog-group-2432013970            0/1     Completed           0          4s

NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/armada-operator    ClusterIP   10.98.2.253    <none>        8383/TCP   57s
service/blog-1-testchart   ClusterIP   10.104.240.7   <none>        80/TCP     6s
service/kubernetes         ClusterIP   10.96.0.1      <none>        443/TCP    7m43s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/armada-operator    1/1     1            1           60s
deployment.apps/blog-1-testchart   0/1     1            0           6s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/armada-operator-cbbc7d7f7     1         1         1       60s
replicaset.apps/blog-1-testchart-5dd8c474f4   1         1         0       6s

Later both charts have been installed

NAME                                    READY   STATUS      RESTARTS   AGE
pod/armada-operator-cbbc7d7f7-zxj5n     1/1     Running     0          44m
pod/blog-1-testchart-5dd8c474f4-26574   1/1     Running     0          43m
pod/blog-2-testchart-57f86dd9c5-xmhd7   1/1     Running     0          43m
pod/wf-blog-group-1193326311            0/1     Completed   0          43m
pod/wf-blog-group-2026876860            0/1     Completed   0          43m
pod/wf-blog-group-2234690393            0/1     Completed   0          43m
pod/wf-blog-group-2432013970            0/1     Completed   0          43m

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/armada-operator    ClusterIP   10.98.2.253      <none>        8383/TCP   44m
service/blog-1-testchart   ClusterIP   10.104.240.7     <none>        80/TCP     43m
service/blog-2-testchart   ClusterIP   10.110.160.242   <none>        80/TCP     43m
service/kubernetes         ClusterIP   10.96.0.1        <none>        443/TCP    51m

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/armada-operator    1/1     1            1           44m
deployment.apps/blog-1-testchart   1/1     1            1           43m
deployment.apps/blog-2-testchart   1/1     1            1           43m

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/armada-operator-cbbc7d7f7     1         1         1       44m
replicaset.apps/blog-1-testchart-5dd8c474f4   1         1         1       43m
replicaset.apps/blog-2-testchart-57f86dd9c5   1         1         1       43m

argo is tracing the steps take to sequence the deployment of the charts:

argo get wf-blog-group

Name:                wf-blog-group
Namespace:           default
ServiceAccount:      armada-argo-sa
Status:              Succeeded
Created:             Fri Mar 22 10:27:28 -0500 (26 seconds ago)
Started:             Fri Mar 22 10:27:28 -0500 (26 seconds ago)
Finished:            Fri Mar 22 10:27:42 -0500 (12 seconds ago)
Duration:            14 seconds

STEP                  PODNAME                   DURATION  MESSAGE
 ✔ wf-blog-group
 ├---✔ enable-blog-1  wf-blog-group-1193326311  2s
 ├---✔ blog-1-ready   wf-blog-group-2432013970  3s
 ├---✔ enable-blog-2  wf-blog-group-2026876860  3s
 └---✔ blog-2-ready   wf-blog-group-2234690393  3s

We can check the state of the ArmadaChart

kubectl get act

NAME     STATE      TARGET STATE   SATISFIED
blog-1   deployed   deployed       true
blog-2   deployed   deployed       true

Run the cleanup

kubectl delete -f examples/argo

examples/sequenced

In that directory, the ArmadaChart (act) are disabled by default and the charts not installed automatically when the act CR are created. When the "ArmadaChartGroup" CR is created, the "chartgroup controller" receives an event and it orchestrate the order of deployment/enablement of the ArmadaChart. The ArmadaChartGroup also becomes owner of the ArmadaChart.

This is basically the same sequencing that above except that it is implemented using an ArmadaChartGroup and an ArmadaManifest

examples/backup

This directory contains the CR definitions involved during an ArmadaBackup procedure.

• WIP

examples/restore

This directory contains the CR definitions involved during an ArmadaRestore procedure.

• WIP

Appendix

POCs contains additional notes regarding successful and failed attempts.