Skip to content

Commit

Permalink
fixup! Update baseline cluster security (#475)
Browse files Browse the repository at this point in the history 
Signed-off-by: Hannes Baum <[email protected]>
  • Loading branch information
@cah-hbaum
cah-hbaum committed May 21, 2024
1 parent cd76a26 commit 6805ea0
Showing 1 changed file with 35 additions and 30 deletions.
65 changes: 35 additions & 30 deletions Standards/scs-0217-v1-cluster-hardening.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ client communication (and therefore cluster communication).
Additionally, HTTPS should be used as the URL schema.
It is also possibly to use a separate CA for the etcd in order to separate and better control access through client
certificates, since etcd by default trusts all the certificates issued by the root CA [2][nsa-cisa].
More information about authentication via TLS are provided in the chapter [ACL restrictions](#acl-restrictions).
More information about authentication via TLS is provided in the chapter [ACL restrictions](#acl-restrictions).

### Securing endpoints

Expand All @@ -123,7 +123,7 @@ ports and try to attack them. In order to minimize the attack vector, internal p
should not be accessible from external networks, except if there are requirements to enable this behavior.

A good way to restrict access would be a combination of firewalls with port
blocking possible network separation.
blocking and the integration of network separation.
How this is done is highly dependent on the specific setup of the provider.
An additional document could be provided in the future to give basic
guidelines for this task.
Expand All @@ -132,22 +132,24 @@ A list of the default ports used in Kubernetes as well as the components accessi

#### Control plane nodes

| Ports | Protocol | Purpose | Used by | Type |
|-----------|----------|-------------------------|----------------------|--------------------|
| 6443 | TCP | API server | All | External, internal |
| 2379-2380 | TCP | etcd server | kube-apiserver, etcd | Internal |
| 10250 | TCP | Kubelet API | Self, Control plane | Internal |
| 10255 | TCP | Read-only Kubelet API | (Monitoring) | External, Internal |
| 10257 | TCP | kube-controller-manager | Self | Internal |
| 10259 | TCP | kube-scheduler | Self | Internal |
| Ports | Protocol | Purpose | Used by | Access type |
|-----------|----------|-------------------------|-----------------------|--------------------|
| 6443 | TCP | API server | All | External, internal |
| 2379-2380 | TCP | etcd server | kube-apiserver, etcd | Internal |
| 10250 | TCP | Kubelet API | Self, Control plane | Internal |
| 10255 | TCP | Read-only Kubelet API | External applications | External, Internal |
| 10257 | TCP | kube-controller-manager | Self | Internal |
| 10259 | TCP | kube-scheduler | Self | Internal |

Hint: `Self` in the `Used by` context means, that a resource will access its own port for requests.

#### Worker nodes

| Ports | Protocol | Purpose | Used by | Type |
|-------------|----------|-----------------------|---------------------|--------------------|
| 10250 | TCP | Kubelet API | Self, Control plane | Internal |
| 10255 | TCP | Read-only Kubelet API | (Monitoring) | External, internal |
| 30000-32767 | TCP | NodePort Services | | External |
| Ports | Protocol | Purpose | Used by | Access type |
|-------------|----------|-----------------------|-----------------------|--------------------|
| 10250 | TCP | Kubelet API | Self, Control plane | Internal |
| 10255 | TCP | Read-only Kubelet API | External applications | External, internal |
| 30000-32767 | TCP | NodePort Services | | External |

### API security, authentication and authorization

Expand All @@ -157,7 +159,7 @@ Access control is applied to both human users and Kubernetes service accounts, w
several stages after a request reaches the API.

1. The Kubernetes API server listens on port 6443 on the first non-localhost network interface by default,
protected by TLS. The TLS certificate can either be signed with a private CA or based on a public key
protected by TLS [3][controlling-access]. The TLS certificate can either be signed with a private CA or based on a public key
infrastructure with a widely recognized CA behind it.
2. The authentication step checks the request for correct authentication based on different possible
authentication modules like password, plain tokens or JWT. Only one of these methods needs to succeed
Expand All @@ -181,7 +183,7 @@ the following (a more complete or up-to-date list may be found in the [Kubernete

This method reads bearer tokens from requests and checks them against a CSV file provided to Kubernetes containing
three columns named `token`, `username` and `uid`. These tokens last indefinitely and the list cant be changed without a
restart of the API server. This makes this option unsuitable for productive clusters.
restart of the API server. This makes this option unsuitable for production clusters.

- *Service Account Tokens*

Expand All @@ -192,7 +194,7 @@ the following (a more complete or up-to-date list may be found in the [Kubernete
pods through the `ServiceAccount` admission controller. Tokens are signed JSON Web Tokens,
that can be used as a Bearer Token or mounted into the pods for API server access.
Since Service Account Tokens are mainly used to allow workloads accessing the API server,
they're not really intended to authenticate users in productive clusters.</p>
they're not really intended to authenticate users in production clusters.

- *X509 client certificates*

Expand Down Expand Up @@ -225,7 +227,7 @@ What will be used by your organization depends on the present setup and the use

Authorization is done after the authentication step in order to check the rights
of a user within the system. Kubernetes authorizes API requests with the API server,
which evaluates requests against all policies in place and then allow or denies these requests.
which evaluates requests against all policies in place and then allows or denies these requests.
By default, a request would be denied.

Kubernetes provides several authentication modes to authorize a request:
Expand Down Expand Up @@ -281,8 +283,8 @@ One recommended admission controller is the [`NodeRestriction` controller][node-
which limits the `Node` and `Pod` objects a Kubelet is allowed to modify to their own `Node` or
objects that are bound to them. It also disallows updating or removing taints and prevents changing
or adding labels with a `node-restriction.kubernetes.io/` prefix.
Be aware that Kubelets will only be limited by this admission controller, if the use credentials
in the `system:nodes` group with a `system:node:<nodeName>` username. Administrators must therefore
Be aware that Kubelets will only be limited by this admission controller, if the user credentials
in the `system:nodes` group begin with a `system:node:<nodeName>` username. Administrators must therefore
configure their Kubelets correctly, if the `NodeRestriction` controller should be fully functional.

### Kubelet access control
Expand All @@ -298,13 +300,14 @@ the health status of the Kubelet.
This port is still available, but it is planned to be [removed][ro-port-removal]
in a future version. At the moment, the port is disabled by default since [Kubernetes 1.10][ro-port-disabled]
and shortly later also in [`kubeadm`][ro-port-disabled-kubeadm].
Different sources recommend disabling this port [3][ro-port-s1] [4][ro-port-s2] due to possible
Different sources recommend disabling this port [4][ro-port-s1] [5][ro-port-s2] due to possible
security risks, but since this standard recommends restricting accessibility of internal ports,
this port wouldn't be accessible from external networks.
It is nevertheless recommended to keep this port disabled, since Kubernetes also acknowledged
its risks and plans to remove it.

By default, requests to the HTTPS endpoint that are not rejected by other authentication
By default, the API server does not verify the Kubelets serving certificate and
requests to the HTTPS endpoint that are not rejected by other authentication
methods are treated as anonymous requests with the combination of name `system:anonymous`
and group `system:unauthenticated`.
This can be disabled by starting the Kubelet with the flag `--anonymous-auth=false`,
Expand Down Expand Up @@ -403,16 +406,17 @@ certificates (since normal username-password-authentication isn't implemented fo
Kubernetes' endpoints MUST be secured in order to provide a small attack surface for bad actors.
It MUST NOT be possible to access Kubernetes ports from outside the internal network hosting the
Kubernetes cluster except for the ports of the API server (default 6443) and the NodePort Services
(default 30000-32767). It is also possible to give access to the read-only Kubelet API port (default 10255),
which is mostly used for monitoring, but we advise against this, mainly because the port is HTTP-only
and can deliver sensitive information to the outside. It is RECOMMENDED to disable this port altogether,
if it isn't in use. Endpoints MUST be secured via HTTPS.
(default 30000-32767). The read-only Kubelet API port (default 10255), which is mostly used for monitoring,
SHOULD be disabled altogether if it isn't in use, mainly because the port is HTTP-only
and can deliver sensitive information to the outside.
Endpoints MUST be secured via HTTPS.

Securing Kubernetes via authentication and authorization is another important topic here.
Authentication is possible through multiple mechanisms, including Kubernetes-provided systems as well as external
authentication processes.
A cluster MUST implement at least two methods for authentication. One of these MUST be *Service Account Tokens*, in order
to provide full functionality to Pods. A second authentication mechanisms can be chosen depending
to provide full functionality to Pods. A second authentication mechanisms can be chosen depending on the requirements
of the provider and/or customer.

Authorization also can be provided through multiple mechanisms.
A cluster MUST activate at least two authorization methods, one of which MUST be *Node authorization* and another one
Expand All @@ -421,13 +425,13 @@ We RECOMMEND RBAC due to it fitting most use cases and being very well documente

In order to harden Kubelet access control, a Kubelet SHOULD only be accessible internally via HTTPS. This is already the
case for the Kubelet API, but the read-only port is only available as HTTP. It is up to the provider to decide if this
read-only port is required or if it can be deactivated (do we recommend this?).
read-only port is required or if it can be deactivated.
Kubelets MUST disable anonymous request authentication to disallow non-rejected requests to go through as anonymous requests.
OPTIONALLY, X509 client certificate authentication or API bearer token authentication can be enabled.
Request authorization for the Kubelet MUST be delegated to the API server via `Webhook` authorization as it is recommended
by the [Kubernetes documentation][kubelet-auth].
Additionally, the `NodeRestriction` admission controller MUST be activated in order to limit interactions between
different Kubelets by disallowing modification of non-bound.
different Kubelets by disallowing modification of `Pod` objects, if they're not bound to the Kubelet requesting the modification.

At last, *Pod security standards* in the form of policies MUST be activated for the cluster. The SCS REQUIRES at least
the *Baseline* policy with the *Restricted* policy CAN also be used.
Expand Down Expand Up @@ -460,6 +464,7 @@ Conformance Tests will be written within another issue.
[openidconnect]: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens
[webhook-token]: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication
[authenticating-proxy]: https://kubernetes.io/docs/reference/access-authn-authz/authentication/#authenticating-proxy
[controlling-access]: https://kubernetes.io/docs/concepts/security/controlling-access/

[ro-port-removal]: https://github.com/kubernetes/kubernetes/issues/12968
[ro-port-disabled]: https://github.com/kubernetes/kubernetes/pull/59666
Expand Down

0 comments on commit 6805ea0

Please sign in to comment.