HelmRelease notification reports install success before helm test results

[salamkar]

Describe the bug

Alerts HelmRelease install success is sent before helm tests results when enabled.

Steps to reproduce

enable a failure in tests for a helm chart.

enable test in on the HelmRelease manifest

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: sample-app
  namespace: flux-demo
spec:
  interval: 1m
  test:
    enable: true
  timeout: 1m
  upgrade:
    remediation:
      remediateLastFailure: true
      strategy: rollback
  chart:
    spec:
      chart: sample-app
      version: 0.2.0
      sourceRef:
        kind: HelmRepository
        name: helmrepo
      interval: 1m

setup a webhook provider

apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Provider
metadata:
  name: gitops-notification
  namespace: flux-demo
spec:
  type: generic
  # webhook address (ignored if secretRef is specified)
  address: http://alert-webhook.default.svc/notify

---
apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Alert
metadata:
  name: demo-alert
  namespace: flux-demo
spec:
  summary: kind-cluster
  providerRef:
    name: gitops-notification
  eventSeverity: info
  eventSources:
    - kind: HelmRelease
      name: '*'

Expected behavior

HelmRelease notification for install success/failure should depend on test results when tests are enabled

Screenshots and recordings

{'involvedObject': {'kind': 'HelmRelease', 'namespace': 'flux-demo', 'name': 'sample-app', 'uid': '901243a4-e26c-4519-ada7-0f4d20dd307c', 'apiVersion': 'helm.toolkit.fluxcd.io/v2beta1', 'resourceVersion': '14400'}, 'severity': 'info', 'timestamp': '2023-06-22T15:41:53Z', 'message': "HelmChart 'flux-demo/flux-demo-sample-app' is not ready", 'reason': 'info', 'metadata': {'summary': 'kind-cluster'}, 'reportingController': 'helm-controller', 'reportingInstance': 'helm-controller-959cbbcf9-f86zr'}

{'involvedObject': {'kind': 'HelmRelease', 'namespace': 'flux-demo', 'name': 'sample-app', 'uid': '901243a4-e26c-4519-ada7-0f4d20dd307c', 'apiVersion': 'helm.toolkit.fluxcd.io/v2beta1', 'resourceVersion': '14409'}, 'severity': 'info', 'timestamp': '2023-06-22T15:41:57Z', 'message': 'Helm install has started', 'reason': 'info', 'metadata': {'summary': 'kind-cluster', 'token': '3675c6a009f8e866b63c9b7cf13aa6de339378f3'}, 'reportingController': 'helm-controller', 'reportingInstance': 'helm-controller-959cbbcf9-f86zr'}

{'involvedObject': {'kind': 'HelmRelease', 'namespace': 'flux-demo', 'name': 'sample-app', 'uid': '901243a4-e26c-4519-ada7-0f4d20dd307c', 'apiVersion': 'helm.toolkit.fluxcd.io/v2beta1', 'resourceVersion': '14409'}, 'severity': 'info', 'timestamp': '2023-06-22T15:41:59Z', 'message': 'Helm install succeeded', 'reason': 'info', 'metadata': {'summary': 'kind-cluster', 'token': '3675c6a009f8e866b63c9b7cf13aa6de339378f3'}, 'reportingController': 'helm-controller', 'reportingInstance': 'helm-controller-959cbbcf9-f86zr'}

{'involvedObject': {'kind': 'HelmRelease', 'namespace': 'flux-demo', 'name': 'sample-app', 'uid': '901243a4-e26c-4519-ada7-0f4d20dd307c', 'apiVersion': 'helm.toolkit.fluxcd.io/v2beta1', 'resourceVersion': '14409'}, 'severity': 'error', 'timestamp': '2023-06-22T15:42:59Z', 'message': 'reconciliation failed: Helm test failed: 1 error occurred:\n\t* timed out waiting for the condition\n\n', 'reason': 'error', 'metadata': {'revision': '0.2.0', 'summary': 'kind-cluster'}, 'reportingController': 'helm-controller', 'reportingInstance': 'helm-controller-959cbbcf9-f86zr'}

{'involvedObject': {'kind': 'HelmRelease', 'namespace': 'flux-demo', 'name': 'sample-app', 'uid': '901243a4-e26c-4519-ada7-0f4d20dd307c', 'apiVersion': 'helm.toolkit.fluxcd.io/v2beta1', 'resourceVersion': '14409'}, 'severity': 'error', 'timestamp': '2023-06-22T15:42:59Z', 'message': 'Helm test failed: 1 error occurred:\n\t* timed out waiting for the condition\n\n\n\nLast Helm logs:\n\n', 'reason': 'error', 'metadata': {'summary': 'kind-cluster', 'token': '3675c6a009f8e866b63c9b7cf13aa6de339378f3'}, 'reportingController': 'helm-controller', 'reportingInstance': 'helm-controller-959cbbcf9-f86zr'}

2023-06-22 15:49:21 webhook-handler INFO: flux_notification: Received: {'involvedObject': {'kind': 'HelmRelease', 'namespace': 'flux-demo', 'name': 'sample-app', 'uid': '901243a4-e26c-4519-ada7-0f4d20dd307c', 'apiVersion': 'helm.toolkit.fluxcd.io/v2beta1', 'resourceVersion': '15059'}, 'severity': 'error', 'timestamp': '2023-06-22T15:49:21Z', 'message': 'reconciliation failed: install retries exhausted', 'reason': 'error', 'metadata': {'summary': 'kind-cluster', 'token': '3675c6a009f8e866b63c9b7cf13aa6de339378f3'}, 'reportingController': 'helm-controller', 'reportingInstance': 'helm-controller-959cbbcf9-f86zr'}

OS / Distro

N/A

Flux version

flux: v2.0.0-rc.4

Flux check

► checking prerequisites
✔ Kubernetes 1.26.3 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.34.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.0.0-rc.4
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.0.0-rc.4
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.0.0-rc.4
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta2
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta2
✔ receivers.notification.toolkit.fluxcd.io/v1
✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[hiddeco]

This is expected behavior as a helm install is a different action than a helm test.

[salamkar]

yes I agree, they are 2 separate actions, but it would be better if the controller wait for the test response (when test is enabled) before sending an install success notification for HelmRelease resources. This way a CI pipeline that relies on alerts can avoid a false positive notification.

is the expected behavior to mimic helm commands?

when using helm commands a failed test doesn't change the helm status to failed.

`helm install sample-app sample-app/
NAME: sample-app
LAST DEPLOYED: Fri Jun 23 09:16:08 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=sample-app,app.kubernetes.io/instance=sample-app" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT`

helm ls
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART          APP VERSION
sample-app      default         1               2023-06-23 09:16:08.014265173 +0100 IST deployed        sample-app-0.2.1.16.0

helm test sample-app
NAME: sample-app
LAST DEPLOYED: Fri Jun 23 09:16:08 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE:     sample-app-test-connection
Last Started:   Fri Jun 23 09:16:33 2023
Last Completed: Fri Jun 23 09:18:53 2023
Phase:          Failed
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=sample-app,app.kubernetes.io/instance=sample-app" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
Error: pod sample-app-test-connection failed

helm ls
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
sample-app      default         1               2023-06-23 09:16:08.014265173 +0100 IST deployed        sample-app-0.2.0        1.16.0

perhaps an improvement for helm notification where the install success message could be used the HelmChart instead of the HelmRelease.
eg. message: "HelmChart 'flux-demo/flux-demo-sample-app install succeded"

but for the "HelmRelease install success" to be an aggregate of the helm chart install and the helm test result.