kwok and k8s python client authentication

[orudi]

Hi! What would be the easyest way to authenticate the k8s python client with kwok cluster api for several clusters?
The only working way that I found is to copy admin.crt admin.key from cluster pki folder and use python client config like this

from kubernetes import client

configuration = client.Configuration()
configuration.host = "hostname:port"
configuration.cert_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), "pki/admin.crt")
configuration.key_file = os.path.join(os.path.dirname(os.path.abspath(__file__)), "pki/admin.key")
client.Configuration.set_default(configuration)
v1 = client.CoreV1Api()
ret = v1.list_pod_for_all_namespaces(watch=False)

so this works, but it requires to get crt and key file every time for each new cluster,
I have tried to use default serviceaccount in order to authenticate with token, but I found that there are no tokens within default serviceaccount, and I can't create a custom token to use it for authentication like this

configuration.api_key["authorization"] = '<bearer_token>'
configuration.api_key_prefix['authorization'] = 'Bearer'

[wzshiming]

We can export kubeconfig with the following command

kwokctl get kubeconfig --name cluster > kubeconfig

then use the exported kubeconfig

examples

from kubernetes import client, config

config.load_kube_config("kubeconfig")

[wzshiming]

I have tried to use default serviceaccount in order to authenticate with token, but I found that there are no tokens within default serviceaccount, and I can't create a custom token to use it for authentication like this

The SA will generate when the cluster is created with --kube-authorization

[wzshiming]

https://kubernetes.io/docs/concepts/configuration/secret/#service-account-token-secrets

In the 1.24 and later, tokens are not generated by default, you need to generate manually with kubectl create token.

[orudi]

~ # kwokctl kubectl apply -f ./kwok_mount_dir/service_account.yaml
serviceaccount/myserviceaccount created
~ # kwokctl kubectl get serviceaccounts
NAME               SECRETS   AGE
default            0         3m45s
myserviceaccount   0         23s

~ # kwokctl kubectl apply -f ./kwok_mount_dir/secret.yaml
secret/mysecret created

~ # kwokctl kubectl describe secret mysecret
Name:         mysecret
Namespace:    default
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: myserviceaccount

Type:  kubernetes.io/service-account-token

Data
====

~ # kwokctl kubectl get serviceaccounts
NAME               SECRETS   AGE
default            0         8m57s
myserviceaccount   0         5m35s

~ # kwokctl kubectl create token myserviceaccount
eyJhbGciOiJSUzI1NiIsImtpZCI6ImRwZ2JoT2FNRVptSkZPRk95QVZac1ZVbHZkQms4Z3ZxdlF5eUNjdzRvNncifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjgwNjI0NzcxLCJpYXQiOjE2ODA2MjExNzEsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VydmljZWFjY291bnQiOnsibmFtZSI6Im15c2VydmljZWFjY291bnQiLCJ1aWQiOiJkNjhkMWIzYy03NTY2LTRkMjctOGJmNi03ODJkYTJkMzE3NjEifX0sIm5iZiI6MTY4MDYyMTE3MSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6bXlzZXJ2aWNlYWNjb3VudCJ9.MSUIKpEz1rx4ctcFVSq0SXlTcIE-B0DKdx7b4DEMsu8SHkazkMLhahopJ5j41L4lczb2ykho6bLKNavVLG3giIAf8q6CObrs8ORxZFsLcDDCawoGuFQ6jyDPakjXr59-zEVpDxrCo4d5X6fjGK5yeyqlP3nx4yE6iiXfyQl3usH8hO1wZpGdlKSebsqmaZfh9VlTgJCCA3CkPMOKZCdZKCYqzzo1um4d1TU9dnNFXw2W6eH6JKk37Bt-lZxej-UZ8RlbwX5KGDUrquWjnjGgUv2dbATEAib8WDodM08E7SlD4ZqjaijBuan-iYkCzrDIMEumH4cVDoPaP8mYITaWyg

~ # kwokctl kubectl get sa
NAME               SECRETS   AGE
default            0         12m
myserviceaccount   0         9m32s

~ # kwokctl kubectl create token myserviceaccount --bound-object-kind='Secret' --bound-object-name='mysecret'
eyJhbGciOiJSUzI1NiIsImtpZCI6ImRwZ2JoT2FNRVptSkZPRk95QVZac1ZVbHZkQms4Z3ZxdlF5eUNjdzRvNncifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjgwNjI0ODYxLCJpYXQiOjE2ODA2MjEyNjEsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VjcmV0Ijp7Im5hbWUiOiJteXNlY3JldCIsInVpZCI6ImVlMGQ2YTcwLWU5ZTctNDQ3ZC05ZDY3LWQxNTFlYjFhZDk2OSJ9LCJzZXJ2aWNlYWNjb3VudCI6eyJuYW1lIjoibXlzZXJ2aWNlYWNjb3VudCIsInVpZCI6ImQ2OGQxYjNjLTc1NjYtNGQyNy04YmY2LTc4MmRhMmQzMTc2MSJ9fSwibmJmIjoxNjgwNjIxMjYxLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpteXNlcnZpY2VhY2NvdW50In0.sKt5nC7CZx8RVBwQNWkiDsGYSHQOujCZDafFUWRyXd2IjsXwcm7dZ0wZQbWSZ9DW4HwOGfWtBF3VGE_J4UHlqMX2A_bYxWGeh7R7bPiVBGXAk_7RatpHdtfEc_-DH8UBhJ0KyAEBi9OIiXYgGESHtJRAH05cuhmB4tqaTTC_QuDyvynvfB7flrYNsC1CWeaywpMpASsOKDxRnXwTtCt63LRhf6zaKC7gvrlQgPyIQFvlIfsSGfL2tpffvXtAr8rqj9jT5RFDx68TlOL65Ndxu8mF7-pR_DIvBU_DQXjCo-0kLhOyNcSBW1u4fN6hYfbLrKlLeg_

~ # kwokctl kubectl get sa
NAME               SECRETS   AGE
default            0         14m
myserviceaccount   0         10m

so, as you can see, no token is bound to service account at the end

[wzshiming]

Yes, this is a major behavioral change in k8s 1.24

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.24.md#no-really-you-must-read-this-before-you-upgrade

The LegacyServiceAccountTokenNoAutoGeneration feature gate is beta, and enabled by default. When enabled, Secret API objects containing service account tokens are no longer auto-generated for every ServiceAccount. Use the TokenRequest API to acquire service account tokens, or if a non-expiring token is required, create a Secret API object for the token controller to populate with a service account token by following this guide. (kubernetes/kubernetes#108309, @zshihang)

[orudi]

Create a service account and associated token:

kwokctl --name {cluster_name} kubectl create serviceaccount my-reader
kwokctl --name {cluster_name} kubectl create token my-reader --duration=8760h