crds/restores.yaml is a invalid yaml format file

[iamjackhu]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
When helm install velero, it will failed with 'http2: client connection lost'.
if use 'kubectl apply -f crds/restores.yaml', it also failed.

What did you expect to happen:
Velero should installed successfully on cluster.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl -n velero apply -f restores.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
error: error when creating “restores.yaml”: Post “https://10.10.10.21:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions?fieldManager=kubectl-client-side-apply”: http2: client connection lost

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
correct the json format of crds/restores.yaml
The invalid line from 205.

Environment:
kubernetes version: 1.21.2

• helm version (use helm version):
• helm chart version and app version (use helm list -n <YOUR NAMESPACE>):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

[iamjackhu]

Is it possible that the file crds/restores.yaml too big?

[jenting]

Could you try this one https://github.com/vmware-tanzu/velero/blob/main/config/crd/v1/bases/velero.io_restores.yaml?

[iamjackhu]

Could you try this one https://github.com/vmware-tanzu/velero/blob/main/config/crd/v1/bases/velero.io_restores.yaml?

Try to do kubectl apply -f it, but failed.
And we logs the api-server, the following is the error message:
I0513 02:06:53.290651 1 client.go:360] parsed scheme: "passthrough" I0513 02:06:53.290973 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379/ <nil> 0 <nil>}] <nil> <nil>} I0513 02:06:53.291051 1 clientconn.go:948] ClientConn switching balancer to "pick_first" E0513 02:07:16.773448 1 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout E0513 02:07:16.774094 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout E0513 02:07:16.775425 1 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout I0513 02:07:16.776996 1 trace.go:205] Trace[1983387252]: "Create" url:/apis/apiextensions.k8s.io/v1/customresourcedefinitions,user-agent:kubectl/v1.23.4 (linux/amd64) kubernetes/e6c093d,client:172.20.10.83,accept:application/json,protocol:HTTP/2.0 (13-May-2022 02:06:24.797) (total time: 51978ms): Trace[1983387252]: ---"About to convert to expected version" 51960ms (02:07:00.758) Trace[1983387252]: [51.97873252s] [51.97873252s] END

And yesterday, kubectl create -f crds/restores.yaml, it will success.

[jenting]

I see, would you mind try kubeadm or k3s or other Kubernetes distro with the same version 1.21.2.

[iamjackhu]

I think the kubectl client version is not the root cause. Because we refer the helm client lib to deploy Velero helm chart to target cluster in our program.

The following is our program's log
"level":"error","timestamp":"2022-05-10T10:15:08.522Z","caller":"rest/api_utils.go:32","msg":"Creating instance with creation error.","reqID":"834d70c2-6ba1-4651-a0ca-c5a9a520a28c","error":"failed to install CRD crds/restores.yaml: Post \"https://192.168.100.19:6443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions?fieldManager=ose_k8s-br\": http2: client connection lost"

[iamjackhu]

I research the Transport code of golang. When TLS connection has no response in 15 seconds, the client side will close the connection and return 'http2: client connection lost' error.

So for this case, the cluster will blocked when install CRD stores.yaml, then kubectl or helm client will return error.

[jenting]

I think there must be something wrong with the restores.yaml manifest on Kubernetes 1.21.2, that why I encourage you to try the other Kubernetes distros or different Kubernetes versions to see if it's reproducible.

The reproduce steps to should be like this

kubectl apply -f https://raw.githubusercontent.com/vmware-tanzu/velero/v1.8.1/config/crd/v1/bases/velero.io_restores.yaml

Correct?

[iamjackhu]

I think there must be something wrong with the restores.yaml manifest on Kubernetes 1.21.2, that why I encourage you to try the other Kubernetes distros or different Kubernetes versions to see if it's reproducible.

The reproduce steps to should be like this

kubectl apply -f https://raw.githubusercontent.com/vmware-tanzu/velero/v1.8.1/config/crd/v1/bases/velero.io_restores.yaml

Correct?

yes

[iamjackhu]

Not all the cluster will meet this issue. It happens on our customer's cluster, which version is 1.21.2

And I wonder why the crd file stores.yaml is invalid yaml format file, excluding it's too big 111K.

Do you know the difference between kubectl apply and create? For the issue cluster, kubectl create works, but kubectl apply will report this error.

[jenting]

Do you know the difference between kubectl apply and create? For the issue cluster, kubectl create works, but kubectl apply will report this error.

Ref to https://www.theserverside.com/answer/Kubectl-apply-vs-create-Whats-the-difference

If the resource doesn't exist, Kubernetes prefers to use create. If the resource existed, use apply instead.
In general, using apply and create for non-existing resource should be the same.
But in your case, it seems no?

[jenting]

Probably related to vmware-tanzu/velero#5089

Handler timeout

Related to vmware-tanzu/velero#4916

[jenting]

We have decreased the crds/restores.yaml size in v1.10 version, closing this issue.