Pod with high priority class can't preempt pod with lower priority class

[kunZooberg]

I created one node with 2 core capacity and 2 pods, one is configure with 100000 priority class, another pod is configured by -1 priority class, at first I apply low priority class pod, then apply the high priority class pod, the expected behavior is a preemption happens, but I still can see the high priority class pod is in pending status.

I created kwok cluste with "kwokctl create cluster --name=kwok"

Pods spec:

`
apiVersion: v1
kind: Pod
metadata:
  name: non-critical-pod2
  labels:
    env: priorityclass-pod
spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      requests:
        memory: "100Mi"
        cpu: "1"
      limits:
        memory: "20Gi"
        cpu: "1"
  priorityClassName: low-priority
  nodeSelector:
    kubernetes.io/os: linux

`

apiVersion: v1
kind: Pod
metadata:
  name: critical-pod
  labels:
    env: priorityclass-pod
spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      requests:
        memory: "100Mi"
        cpu: "1"
      limits:
        memory: "20Gi"
        cpu: "1"
  priorityClassName: critical-priority
  nodeSelector:
    kubernetes.io/os: linux

Node:

apiVersion: v1
kind: Node
metadata:
  annotations:
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
    kwok.x-k8s.io/node: fake
  creationTimestamp: "2023-04-26T15:13:50Z"
  labels:
    agentpool: lp014
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/instance-type: Standard_D2_v3
    beta.kubernetes.io/os: linux
    failure-domain.beta.kubernetes.io/region: southcentralus
    failure-domain.beta.kubernetes.io/zone: "0"
    kubernetes.azure.com/agentpool: lp014
    kubernetes.azure.com/mode: system
    kubernetes.azure.com/node-image-version: AKSCBLMariner-V1-2023.01.26
    kubernetes.azure.com/os-sku: CBLMariner
    kubernetes.azure.com/role: agent
    kubernetes.azure.com/storageprofile: managed
    kubernetes.azure.com/storagetier: Standard_LRS
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: aks-lp014
    kubernetes.io/os: linux
    kubernetes.io/role: agent
    node-role.kubernetes.io/agent: ""
    node.kubernetes.io/instance-type: Standard_D2_v3
    storageprofile: managed
    storagetier: Standard_LRS
    topology.disk.csi.azure.com/zone: ""
    topology.kubernetes.io/region: southcentralus
    topology.kubernetes.io/zone: "0"
  name: aks-lp014
spec:
  providerID: azure
status:
  addresses:
  - address: 10.60.9.163
    type: InternalIP
  - address: aks-lp014
    type: Hostname
  allocatable:
    cpu: 1900m
    ephemeral-storage: "121322059576"
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 5362024Ki
    pods: "100"
  capacity:
    cpu: "2"
    ephemeral-storage: 131642860Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8017256Ki
    pods: "100"
  nodeInfo:
    architecture: amd64
    containerRuntimeVersion: containerd://1.6.6+azure
    kernelVersion: 5.10.164.1-1.cm1
    kubeProxyVersion: fake
    kubeletVersion: fake
    operatingSystem: linux
    osImage: CBL-Mariner/Linux

Priority Class

apiVersion: scheduling.k8s.io/v1
description: high priority class.
kind: PriorityClass
metadata:
  name: critical-priority
preemptionPolicy: PreemptLowerPriority
value: 999998

apiVersion: scheduling.k8s.io/v1
description: low priority class.
kind: PriorityClass
metadata:
  name: low-priority
preemptionPolicy: PreemptLowerPriority
value: -1

[wzshiming]

https://github.com/kubernetes/kubernetes/blob/ccde866a2fefd3963499dfc9873bb9b17bd4855d/pkg/scheduler/framework/plugins/defaultpreemption/default_preemption.go#L166

https://github.com/kubernetes/component-helpers/blob/6f40a0f5eeff05d61c9ee6e21b25fcd1ab2c422d/scheduling/corev1/helpers.go#L26-L35

The preemption priority of the scheduler seems to ignore the priorityClassName, you will need to set priority

  priorityClassName: critical-priority
  priority: 999999

https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
I feel there is a difference between the documentation and the implementation, documentation uses priorityClassName to indicate preemption priority, but implementation is not.

@Huang-Wei

Could you have time to look at it, it feels like a bug in the scheduler but I'm not sure

[kunZooberg]

Thanks @wzshiming's fast response, the same config works in real k8s cluster but failed in kwok.

[kunZooberg]

From https://github.com/kubernetes/api/blob/v0.27.1/core/v1/types.go#L3202

Will kwok enable the corresponding admission controller?

// The priority value. Various system components use this field to find the
// priority of the pod. When Priority Admission Controller is enabled, it
// prevents users from setting this field. The admission controller populates
// this field from PriorityClassName.
// The higher the value, the higher the priority.
// +optional
Priority *int32 json:"priority,omitempty" protobuf:"bytes,25,opt,name=priority"

[wzshiming]

Ohh, that looks like you need to add --kube-admission when creating cluster

kwokctl create cluster --kube-admission

[Huang-Wei]

@wzshiming I think you've already figured it out - the PriorityClassName (string) -> Priory (int) population is done by admission plugin. I'd suggest to enable --kube-admission by default for kwokctl, wdyt? basically try to use the same parameter as default API Server.

@kunZooberg another workaround is to directly specify .spec.priority in pod spec.

[wzshiming]

I agree with you

[kunZooberg]

Thanks for all the responses.