FluxCD keeps pushing deleted tag

[kamikaze]

I had a tag in my infra repo which I did delete recently. But it seems FluxCD still keeps it on its side and pushes on every image automation update. Is there any way to not do that?

[darkowlzz]

Hi @kamikaze , can you please provide some more information to reproduce this issue. I tried reproducing this by trying various scenarios but couldn't reproduce it.
First I created an IUA (ImageUpdateAutomation) with same checkout and push branch. Made some updates and created a few tags for some commits made by IUA. Then I deleted the created tags and allowed IUA to make more updates and none of the old tags appeared again.
Then I changed the push branch to be different from checkout branch and made more update commits. Created tags in both checkout branch and push branch. Made more update commits and deleted the tags, made more update commits, no old tags appeared again.
I tried this on gitlab.

Can you please share more details about your setup and configuration?
As far as I know, image-automation-controller does a targeted push to a specific branch or reference as configured. It shouldn't be able to push anything else. Also, the way image-automation-controller works, it doesn't store any git repository locally. It does a fresh clone of the source from remote in a different temporary directory every time it needs to apply an update and commit. So it doesn't retain any git repository history.
Is it possible that something else in your environment is doing this?

[kingdonb]

Can you be sure that Flux is what recreates the tags? There are architectural reasons why we are pretty sure Flux can't do this. So unless you are specifically testing and can show a sequence of how this happens, I suspect it's something else as @darkowlzz said. There is no code in Flux which pushes tags or pushes all tags.

I do remember running this issue with a person on my team before (not related to Flux) who did not ever run brew upgrade, or could not for some reason, I told them they were running a dangerously old version of git and I don't remember all the details, but they were running git push --tags or git push --force iirc, or maybe git push --tags --force and it was pushing over tags, pushing all tags including any that were not named. This sounds almost like what you described. We had this discussion at the Flux dev meeting today, trying to understand what is happening from such a brief report, we need more information, but maybe you are having the same issue.

I think this was a breaking change in Git, it looks like Git 2.0 changed the semantics of git push so this could not happen so easily. Is there anyone on your team who has access to push tags who might be running Git <2.0? (Are you the only person with access to push tags?)

This is just a guess, but I remember it well even though it was years ago, because it was so confusing. Anyway, the old version of git did not take care to avoid this issue, while the newer version of git cannot so easily push all tags causing the same confusion.