HelmRelease stuck in state "reconciliation in progress"

[runningman84]

We have some HelmRelease objects which are stuck in state: "reconciliation in progress".

The helm release itself looks fine, we just need to reset the state or refresh it... deleting this object would also help but this can cause unwanted side effects.

Name:         efs-csi
Namespace:    kube-system
Labels:       kustomize.toolkit.fluxcd.io/name=efs-csi
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  reconcile.fluxcd.io/requestedAt: 2024-07-22T07:56:33.325266586Z
API Version:  helm.toolkit.fluxcd.io/v2
Kind:         HelmRelease
Metadata:
  Creation Timestamp:  2023-09-27T11:08:59Z
  Finalizers:
    finalizers.fluxcd.io
  Generation:        16
  Resource Version:  1402071395
  UID:               7dc394d7-26f6-4658-a235-f88afe654c00
Spec:
  Chart:
    Spec:
      Chart:               aws-efs-csi-driver
      Interval:            14m
      Reconcile Strategy:  ChartVersion
      Source Ref:
        Kind:       HelmRepository
        Name:       efs-csi
        Namespace:  kube-system
      Version:      3.0.8
  Install:
    Crds:              CreateReplace
    Create Namespace:  true
  Interval:            9m
  Upgrade:
    Crds:  CreateReplace
  Values:
    Controller:
      Affinity:
        Node Affinity:
          Preferred During Scheduling Ignored During Execution:
            Preference:
              Match Expressions:
                Key:       kubernetes.io/arch
                Operator:  In
                Values:
                  arm64
            Weight:  1
          Required During Scheduling Ignored During Execution:
            Node Selector Terms:
              Match Expressions:
                Key:       kubernetes.io/os
                Operator:  In
                Values:
                  linux
      Resources:
        Limits:
          Memory:  512Mi
        Requests:
          Cpu:     100m
          Memory:  128Mi
      Service Account:
        Create:  false
        Name:    aws-efs-csi-driver-oidc
      Tolerations:
        Effect:    NoSchedule
        Key:       arch
        Operator:  Equal
        Value:     arm64
    Node:
      Resources:
        Limits:
          Memory:  512Mi
        Requests:
          Cpu:     10m
          Memory:  128Mi
    Sidecars:
      Csi Provisioner:
        Resources:
          Limits:
            Memory:  64Mi
          Requests:
            Cpu:     10m
            Memory:  32Mi
      Liveness Probe:
        Resources:
          Limits:
            Memory:  64Mi
          Requests:
            Cpu:     10m
            Memory:  16Mi
      Node Driver Registrar:
        Resources:
          Limits:
            Memory:  64Mi
          Requests:
            Cpu:     10m
            Memory:  16Mi
Status:
  Conditions:
    Last Transition Time:  2024-10-31T07:55:35Z
    Message:               Failed to upgrade after 1 attempt(s)
    Observed Generation:   16
    Reason:                RetriesExceeded
    Status:                True
    Type:                  Stalled
    Last Transition Time:  2024-10-31T07:46:21Z
    Message:               reconciliation in progress
    Observed Generation:   16
    Reason:                Progressing
    Status:                Unknown
    Type:                  Ready
    Last Transition Time:  2024-08-12T06:07:01Z
    Message:               Helm upgrade failed for release kube-system/efs-csi with chart [email protected]: client rate limiter Wait returned an error: context deadline exceeded
    Observed Generation:   16
    Reason:                UpgradeFailed
    Status:                False
    Type:                  Released
  Failures:                1
  Helm Chart:              kube-system/kube-system-efs-csi
  History:
    App Version:                  2.0.7
    Chart Name:                   aws-efs-csi-driver
    Chart Version:                3.0.8
    Config Digest:                sha256:5020f3780e9aaca0b60c1669a1d205023e2e2bd83f26d7b8fe157ebaa1af2bf8
    Digest:                       sha256:3431ea19feff087241d6cf02e5565b10f63f796df640674da17f6344854fdf26
    First Deployed:               2023-09-27T11:09:11Z
    Last Deployed:                2024-08-12T06:01:58Z
    Name:                         efs-csi
    Namespace:                    kube-system
    Status:                       failed
    Version:                      15
    App Version:                  2.0.6
    Chart Name:                   aws-efs-csi-driver
    Chart Version:                3.0.7
    Config Digest:                sha256:5020f3780e9aaca0b60c1669a1d205023e2e2bd83f26d7b8fe157ebaa1af2bf8
    Digest:                       sha256:94a0f27f98af952b96272d8f3481ddec92841eb8d91219be4b80093e74ebbdd6
    First Deployed:               2023-09-27T11:09:11Z
    Last Deployed:                2024-07-29T06:02:04Z
    Name:                         efs-csi
    Namespace:                    kube-system
    Status:                       deployed
    Version:                      14
    App Version:                  2.0.5
    Chart Name:                   aws-efs-csi-driver
    Chart Version:                3.0.6
    Config Digest:                sha256:5020f3780e9aaca0b60c1669a1d205023e2e2bd83f26d7b8fe157ebaa1af2bf8
    Digest:                       sha256:05cae22f7eebd326facc9d44d965e3063a738580ed22f58b31b164ae79d0743f
    First Deployed:               2023-09-27T11:09:11Z
    Last Deployed:                2024-07-22T07:49:51Z
    Name:                         efs-csi
    Namespace:                    kube-system
    Status:                       superseded
    Version:                      13
  Last Attempted Config Digest:   sha256:5020f3780e9aaca0b60c1669a1d205023e2e2bd83f26d7b8fe157ebaa1af2bf8
  Last Attempted Generation:      16
  Last Attempted Release Action:  upgrade
  Last Attempted Revision:        3.0.8
  Last Handled Reconcile At:      2024-07-22T07:56:33.325266586Z
  Observed Generation:            16
  Storage Namespace:              kube-system
  Upgrade Failures:               1
Events:                           <none>

[stefanprodan]

There are Flux CLI commands that can help you force & retry the upgrade, see the docs here: https://fluxcd.io/flux/components/helm/helmreleases/#forcing-a-release