From 6805ea0289d5595323c73ed5df8b29d9e90fd889 Mon Sep 17 00:00:00 2001 From: Hannes Baum Date: Tue, 21 May 2024 14:04:52 +0200 Subject: [PATCH] fixup! Update baseline cluster security (#475) Signed-off-by: Hannes Baum --- Standards/scs-0217-v1-cluster-hardening.md | 65 ++++++++++++---------- 1 file changed, 35 insertions(+), 30 deletions(-) diff --git a/Standards/scs-0217-v1-cluster-hardening.md b/Standards/scs-0217-v1-cluster-hardening.md index e4c8eb102..feac84870 100644 --- a/Standards/scs-0217-v1-cluster-hardening.md +++ b/Standards/scs-0217-v1-cluster-hardening.md @@ -112,7 +112,7 @@ client communication (and therefore cluster communication). Additionally, HTTPS should be used as the URL schema. It is also possibly to use a separate CA for the etcd in order to separate and better control access through client certificates, since etcd by default trusts all the certificates issued by the root CA [2][nsa-cisa]. -More information about authentication via TLS are provided in the chapter [ACL restrictions](#acl-restrictions). +More information about authentication via TLS is provided in the chapter [ACL restrictions](#acl-restrictions). ### Securing endpoints @@ -123,7 +123,7 @@ ports and try to attack them. In order to minimize the attack vector, internal p should not be accessible from external networks, except if there are requirements to enable this behavior. A good way to restrict access would be a combination of firewalls with port -blocking possible network separation. +blocking and the integration of network separation. How this is done is highly dependent on the specific setup of the provider. An additional document could be provided in the future to give basic guidelines for this task. @@ -132,22 +132,24 @@ A list of the default ports used in Kubernetes as well as the components accessi #### Control plane nodes -| Ports | Protocol | Purpose | Used by | Type | -|-----------|----------|-------------------------|----------------------|--------------------| -| 6443 | TCP | API server | All | External, internal | -| 2379-2380 | TCP | etcd server | kube-apiserver, etcd | Internal | -| 10250 | TCP | Kubelet API | Self, Control plane | Internal | -| 10255 | TCP | Read-only Kubelet API | (Monitoring) | External, Internal | -| 10257 | TCP | kube-controller-manager | Self | Internal | -| 10259 | TCP | kube-scheduler | Self | Internal | +| Ports | Protocol | Purpose | Used by | Access type | +|-----------|----------|-------------------------|-----------------------|--------------------| +| 6443 | TCP | API server | All | External, internal | +| 2379-2380 | TCP | etcd server | kube-apiserver, etcd | Internal | +| 10250 | TCP | Kubelet API | Self, Control plane | Internal | +| 10255 | TCP | Read-only Kubelet API | External applications | External, Internal | +| 10257 | TCP | kube-controller-manager | Self | Internal | +| 10259 | TCP | kube-scheduler | Self | Internal | + +Hint: `Self` in the `Used by` context means, that a resource will access its own port for requests. #### Worker nodes -| Ports | Protocol | Purpose | Used by | Type | -|-------------|----------|-----------------------|---------------------|--------------------| -| 10250 | TCP | Kubelet API | Self, Control plane | Internal | -| 10255 | TCP | Read-only Kubelet API | (Monitoring) | External, internal | -| 30000-32767 | TCP | NodePort Services | | External | +| Ports | Protocol | Purpose | Used by | Access type | +|-------------|----------|-----------------------|-----------------------|--------------------| +| 10250 | TCP | Kubelet API | Self, Control plane | Internal | +| 10255 | TCP | Read-only Kubelet API | External applications | External, internal | +| 30000-32767 | TCP | NodePort Services | | External | ### API security, authentication and authorization @@ -157,7 +159,7 @@ Access control is applied to both human users and Kubernetes service accounts, w several stages after a request reaches the API. 1. The Kubernetes API server listens on port 6443 on the first non-localhost network interface by default, -protected by TLS. The TLS certificate can either be signed with a private CA or based on a public key +protected by TLS [3][controlling-access]. The TLS certificate can either be signed with a private CA or based on a public key infrastructure with a widely recognized CA behind it. 2. The authentication step checks the request for correct authentication based on different possible authentication modules like password, plain tokens or JWT. Only one of these methods needs to succeed @@ -181,7 +183,7 @@ the following (a more complete or up-to-date list may be found in the [Kubernete This method reads bearer tokens from requests and checks them against a CSV file provided to Kubernetes containing three columns named `token`, `username` and `uid`. These tokens last indefinitely and the list cant be changed without a - restart of the API server. This makes this option unsuitable for productive clusters. + restart of the API server. This makes this option unsuitable for production clusters. - *Service Account Tokens* @@ -192,7 +194,7 @@ the following (a more complete or up-to-date list may be found in the [Kubernete pods through the `ServiceAccount` admission controller. Tokens are signed JSON Web Tokens, that can be used as a Bearer Token or mounted into the pods for API server access. Since Service Account Tokens are mainly used to allow workloads accessing the API server, - they're not really intended to authenticate users in productive clusters.

+ they're not really intended to authenticate users in production clusters. - *X509 client certificates* @@ -225,7 +227,7 @@ What will be used by your organization depends on the present setup and the use Authorization is done after the authentication step in order to check the rights of a user within the system. Kubernetes authorizes API requests with the API server, -which evaluates requests against all policies in place and then allow or denies these requests. +which evaluates requests against all policies in place and then allows or denies these requests. By default, a request would be denied. Kubernetes provides several authentication modes to authorize a request: @@ -281,8 +283,8 @@ One recommended admission controller is the [`NodeRestriction` controller][node- which limits the `Node` and `Pod` objects a Kubelet is allowed to modify to their own `Node` or objects that are bound to them. It also disallows updating or removing taints and prevents changing or adding labels with a `node-restriction.kubernetes.io/` prefix. -Be aware that Kubelets will only be limited by this admission controller, if the use credentials -in the `system:nodes` group with a `system:node:` username. Administrators must therefore +Be aware that Kubelets will only be limited by this admission controller, if the user credentials +in the `system:nodes` group begin with a `system:node:` username. Administrators must therefore configure their Kubelets correctly, if the `NodeRestriction` controller should be fully functional. ### Kubelet access control @@ -298,13 +300,14 @@ the health status of the Kubelet. This port is still available, but it is planned to be [removed][ro-port-removal] in a future version. At the moment, the port is disabled by default since [Kubernetes 1.10][ro-port-disabled] and shortly later also in [`kubeadm`][ro-port-disabled-kubeadm]. -Different sources recommend disabling this port [3][ro-port-s1] [4][ro-port-s2] due to possible +Different sources recommend disabling this port [4][ro-port-s1] [5][ro-port-s2] due to possible security risks, but since this standard recommends restricting accessibility of internal ports, this port wouldn't be accessible from external networks. It is nevertheless recommended to keep this port disabled, since Kubernetes also acknowledged its risks and plans to remove it. -By default, requests to the HTTPS endpoint that are not rejected by other authentication +By default, the API server does not verify the Kubelets serving certificate and +requests to the HTTPS endpoint that are not rejected by other authentication methods are treated as anonymous requests with the combination of name `system:anonymous` and group `system:unauthenticated`. This can be disabled by starting the Kubelet with the flag `--anonymous-auth=false`, @@ -403,16 +406,17 @@ certificates (since normal username-password-authentication isn't implemented fo Kubernetes' endpoints MUST be secured in order to provide a small attack surface for bad actors. It MUST NOT be possible to access Kubernetes ports from outside the internal network hosting the Kubernetes cluster except for the ports of the API server (default 6443) and the NodePort Services -(default 30000-32767). It is also possible to give access to the read-only Kubelet API port (default 10255), -which is mostly used for monitoring, but we advise against this, mainly because the port is HTTP-only -and can deliver sensitive information to the outside. It is RECOMMENDED to disable this port altogether, -if it isn't in use. Endpoints MUST be secured via HTTPS. +(default 30000-32767). The read-only Kubelet API port (default 10255), which is mostly used for monitoring, +SHOULD be disabled altogether if it isn't in use, mainly because the port is HTTP-only +and can deliver sensitive information to the outside. +Endpoints MUST be secured via HTTPS. Securing Kubernetes via authentication and authorization is another important topic here. Authentication is possible through multiple mechanisms, including Kubernetes-provided systems as well as external authentication processes. A cluster MUST implement at least two methods for authentication. One of these MUST be *Service Account Tokens*, in order -to provide full functionality to Pods. A second authentication mechanisms can be chosen depending +to provide full functionality to Pods. A second authentication mechanisms can be chosen depending on the requirements +of the provider and/or customer. Authorization also can be provided through multiple mechanisms. A cluster MUST activate at least two authorization methods, one of which MUST be *Node authorization* and another one @@ -421,13 +425,13 @@ We RECOMMEND RBAC due to it fitting most use cases and being very well documente In order to harden Kubelet access control, a Kubelet SHOULD only be accessible internally via HTTPS. This is already the case for the Kubelet API, but the read-only port is only available as HTTP. It is up to the provider to decide if this -read-only port is required or if it can be deactivated (do we recommend this?). +read-only port is required or if it can be deactivated. Kubelets MUST disable anonymous request authentication to disallow non-rejected requests to go through as anonymous requests. OPTIONALLY, X509 client certificate authentication or API bearer token authentication can be enabled. Request authorization for the Kubelet MUST be delegated to the API server via `Webhook` authorization as it is recommended by the [Kubernetes documentation][kubelet-auth]. Additionally, the `NodeRestriction` admission controller MUST be activated in order to limit interactions between -different Kubelets by disallowing modification of non-bound. +different Kubelets by disallowing modification of `Pod` objects, if they're not bound to the Kubelet requesting the modification. At last, *Pod security standards* in the form of policies MUST be activated for the cluster. The SCS REQUIRES at least the *Baseline* policy with the *Restricted* policy CAN also be used. @@ -460,6 +464,7 @@ Conformance Tests will be written within another issue. [openidconnect]: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens [webhook-token]: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication [authenticating-proxy]: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authenticating-proxy +[controlling-access]: https://kubernetes.io/docs/concepts/security/controlling-access/ [ro-port-removal]: https://github.com/kubernetes/kubernetes/issues/12968 [ro-port-disabled]: https://github.com/kubernetes/kubernetes/pull/59666