CI: Helm chart publishing still broken

[trevorwhitney]

#14799 fixed the helm-ci action, allowing helm PRs to be merged. However, publishing releases is still broken:

• https://github.com/grafana/loki/actions/runs/11727098884/job/32667387047
• https://github.com/grafana/loki/actions/runs/11728642157/job/32672566084

Run helm push "/home/runner/work/loki/loki/.cr-release-packages/loki-6.19.0.tgz" "oci://ghcr.io/${GITHUB_REPOSITORY_OWNER}/helm-charts"
  helm push "/home/runner/work/loki/loki/.cr-release-packages/loki-6.19.0.tgz" "oci://ghcr.io/${GITHUB_REPOSITORY_OWNER}/helm-charts"
  shell: /usr/bin/bash -e {0}
  env:
    CR_CONFIGFILE: /home/runner/work/loki/loki/source/production/helm/cr.yaml
    CT_CONFIGFILE: /home/runner/work/loki/loki/source/production/helm/ct.yaml
    CR_INDEX_PATH: /home/runner/work/loki/loki/.cr-index
    CR_PACKAGE_PATH: /home/runner/work/loki/loki/.cr-release-packages
    CR_TOOL_PATH: /home/runner/work/loki/loki/.cr-tool
    HELM_TAG_PREFIX: helm
    github_app_id: ***
    AUTHTOKEN: ***
Error: unexpected status from POST request to https://ghcr.io/v[2](https://github.com/grafana/loki/actions/runs/11727098884/job/32667387047#step:18:2)/grafana/helm-charts/loki/blobs/uploads/: 403 Forbidden
Error: Process completed with exit code 1.

Last successful run was 3 weeks ago: https://github.com/grafana/loki/actions/runs/11374046600

[trevorwhitney]

@morganrowse if you're still interested, looks like the saga isn't quite over

[morganrowse]

Hi @trevorwhitney just got back from leave and now ready to look into this.

I have a theory that this CI issue was introduced by some upstream changes to grafana/helm-charts shared github actions here, The grafana/loki helm-release action makes use of the main branch here, something akin to using latest on a docker container.

It looks like the core issue here follows this PR grafana/helm-charts#3418

@narqo just raising for visibility if you have not seen this for the Loki chart. Im not seeing any fundemental code differences between mimr and loki CI. Are there GitHub app or repo permission settings required to get this working as expected?

grafana/helm-charts#3068 (comment)

[narqo]

Im not seeing any fundemental code differences between mimr and loki CI. Are there GitHub app or repo permission settings required to get this working as expected?

It should just work since grafana/helm-charts#3424 — no special changes on your side needed (it works for k6-operator also).

[trevorwhitney]

@narqo we're referencing thew workflow @ main but still seeing those 403 errors Error: unexpected status from POST request to https://ghcr.io/v2/grafana/helm-charts/loki/blobs/uploads/: 403 Forbidden

[trevorwhitney]

Awesome, thanks @morganrowse, I reverted the release pipeline to 70dbbb722dee3f2ee126e12684cc0e92a20972ed before the ghcr changes, and our release went out successfully this week. Going to keep this issue open until we fix it for real though, as we should bring back the ghcr pushing.