Support CRIO runtime-based Kubernetes Clusters #418
Comments
|
Hi @veluruchaithanya, I know we allow CRI-O for guest-pull image pulling for instance, and the support was improved here #376. Do you mind sharing the details or errors you are facing? |
|
Hi @beraldoleal, I have installed Red Hat Openshift on the bare-metal server. This is intel server and supports SGX and TDX features. I tried to install CoCo Operator on the Openshift platform using below commands and it is not creating kata runtime classes and also only one pod is up and running. The pods "cc-operator-daemon-install" and "cc-operator-pre-install-daemon" are not installed. I also see same issue in the below open ticket. Here it was mentioned that due to no CRIO support this issue cant be fixed. |
|
@veluruchaithanya thank you. Could you double check if the workernodes has the following label? And also, the logs for manager would be great: |
@ldoktor I have not tried latest CoCo operator with crio support on OCP. If you can try and share the details as mentioned by @beraldoleal in #418 (comment) it'll help. |
|
I have with similar results as @veluruchaithanya. The worker nodes were not labelled things started happening and I got I got Looking at the services it's called The manager logs (from the run without security constrains): but I guess the main issue is the cri-o service name and the node labeling... |
|
OK I went ahead and tried: diff --git a/install/pre-install-payload/scripts/reqs-deploy.sh b/install/pre-install-payload/scripts/reqs-deploy.sh
index 8ae05f0..a497f49 100755
--- a/install/pre-install-payload/scripts/reqs-deploy.sh
+++ b/install/pre-install-payload/scripts/reqs-deploy.sh
@@ -167,8 +167,13 @@ function uninstall_artifacts() {
function restart_systemd_service() {
host_systemctl daemon-reload
- echo "Restarting ${container_engine}"
- host_systemctl restart "${container_engine}"
+ if [ "${container_engine}" == "cri-o" ]; then
+ service_name="crio"
+ else
+ service_name="${container_engine}"
+ fi
+ echo "Restarting ${service_name}"
+ host_systemctl restart "${service_name}"
}
function configure_nydus_snapshotter_for_containerd() {pushed it here And all the expected resources: @bpradipt do you think such change would be acceptable or is it too hackish? Another story will be the security constrains but IIRC it should be only a matter of creating the right labels, shouldn't it? |
|
Nice work @ldoktor .. The |
|
The trick I used is used in e2e testing but it's explicitly marked as DO-NOT-USE-IN-PRODUCTION as it effectively disables the pod security framework. Anyway let me send the reqs-deploy fix and we can experiment with a proper fix of the permissions later. |
the cri-o runtime uses "crio.service" service name (tested on OCP 4.14) Related to: confidential-containers#418 Signed-off-by: Lukáš Doktor <[email protected]>
|
Well the operator already creates a service account so we just need to allow the service account to use privileged operations by If you try using it, you get: This is because one needs to tell selinux that /opt/kata contains runeable binaries. This is pending on kata-containers: kata-containers/kata-containers#8417 and is treated by CI temporarily here https://github.com/kata-containers/kata-containers/blob/main/ci/openshift-ci/cluster/deployments/relabel_selinux.yaml To use it with the operator that deploys things inside confidential-containers-system namespaces you can use a similar approach just changing the namespaces and service accounts: apiVersion: apps/v1
kind: DaemonSet
metadata:
name: relabel-selinux-daemonset
namespace: confidential-containers-system
spec:
selector:
matchLabels:
app: restorecon
template:
metadata:
labels:
app: restorecon
spec:
serviceAccountName: cc-operator-controller-manager
hostPID: true
containers:
- name: relabel-selinux-container
image: alpine
securityContext:
privileged: true
command: ["/bin/sh", "-c", "
set -e;
echo Starting the relabel;
nsenter --target 1 --mount bash -xc '
command -v semanage &>/dev/null || { echo Does not look like a SELINUX cluster, skipping; exit 0; };
for ENTRY in \
\"/(.*/)?opt/kata/bin(/.*)?\" \
\"/(.*/)?opt/kata/runtime-rs/bin(/.*)?\" \
\"/(.*/)?opt/kata/share/kata-.*(/.*)?(/.*)?\" \
\"/(.*/)?opt/kata/share/ovmf(/.*)?\" \
\"/(.*/)?opt/kata/share/tdvf(/.*)?\" \
\"/(.*/)?opt/kata/libexec(/.*)?\";
do
semanage fcontext -a -t qemu_exec_t \"$ENTRY\" || semanage fcontext -m -t qemu_exec_t \"$ENTRY\" ||
{ echo \"Error in semanage command\"; exit 1; }
done;
restorecon -v -R /opt/kata || { echo \"Error in restorecon command\"; exit 1; }
';
echo NSENTER_FINISHED_WITH: $?;
sleep infinity"]Which those changes I can successfully run a sample kata-qemu pod: # Copyright (c) 2020 Red Hat, Inc.
#
# SPDX-License-Identifier: Apache-2.0
#
# Define the pod for a http server app.
---
apiVersion: v1
kind: Pod
metadata:
name: http-server
labels:
app: http-server-app
spec:
containers:
- name: http-server
image: registry.fedoraproject.org/fedora
ports:
- containerPort: 8080
command: ["python3"]
args: [ "-m", "http.server", "8080"]
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
runAsNonRoot: true
runAsUser: 1000
seccompProfile:
type: RuntimeDefault
runtimeClassName: kata-qemu(tested on ocp 4.16 today) Now @bpradipt I know how to make things working but I'm not sure how to add this to operator to make it work out of the box (apart from adding this to the documentation). Do you know if we want to add this and how to do it? |
|
@ldoktor can kata-deploy handle the selinux relabling when installing on systems with selinux enabled? |
That is what is being discussed in kata-containers/kata-containers#8417 (also note that this has nothing to do with the operator, it fails with kata-containers only as well so it needs to be addressed there) |
the cri-o runtime naming is inconsistent and for service uses "crio.service" name (without the "-"). To be consistent with kata-containers override the container_engine variable to "crio" and update all places where it's being used. Related to: confidential-containers#418 Signed-off-by: Lukáš Doktor <[email protected]>
the cri-o runtime naming is inconsistent and for service uses "crio.service" name (without the "-"). To be consistent with kata-containers override the container_engine variable to "crio" and update all places where it's being used. Related to: confidential-containers#418 Signed-off-by: Lukáš Doktor <[email protected]>
the cri-o runtime naming is inconsistent and for service uses "crio.service" name (without the "-"). To be consistent with kata-containers override the container_engine variable to "crio" and update all places where it's being used. Related to: #418 Signed-off-by: Lukáš Doktor <[email protected]>
Hello, I'm looking to deploy Confidential Containers on Red Hat OpenShift to run Linux applications inside SGX and TDX secure enclaves. Unfortunately, I am unable to use the CoCo operator on Red Hat OpenShift, as OpenShift uses CRIO as the container runtime, which is not yet supported by Confidential Containers. Can anyone provide a rough time estimate for when the CoCo operator will support CRIO runtime-based Kubernetes clusters?
The text was updated successfully, but these errors were encountered: