Repository navigation
Conversation
Signed-off-by: Aurelio <19254254+Aureliolo@users.noreply.github.com>
Why should the job be recreated? If the helm command is failing and you're trying to get it to continue running until it succeeds, then you should increase
How exactly would this happen? Helm creates a new release resource before it even starts upgrading, as a way to lock the chart. If that fails then the helm command would fail, and the pod should be restarted as controlled by the backOffLimit - NOT by the job being recreated. I think you're expecting the controller to do more than it should do. |
|
Hi @brandond, thanks for taking the time to look at this and explain how it's meant to work. Yeah, I think I opened this one too early, sorry. You're right that helm creates the release first. The case I hit is helm failing before it even gets there. The easiest way I found to get it is restarting k3s while a retry runs: the chart download from the apiserver gets connection refused, the retry branch still does exit 0, the job completes and Traefik (or whichever chart it is) stays failed until someone deletes the job. backOffLimit never kicks in because the pod never fails. Recreating the job like this PR does would loop forever on stuff that always fails though, so that's not great either. Would it make more sense in klipper-helm? In retry, exit 1 when helm didn't store a new revision, so the kubelet backs off like you described. A failed release that did get stored would still exit 0 like in #140. To be fair I only ran into it because something else broke the release first, so it's not super common. Fine to close this one either way. |
Can you show logs from this? This does sound like an issue in the klipper-helm entry script or perhaps helm cli itself. If there is an error accessing the chart helm (and the script) should exit with a nonzero exit code. |
|
Sure. k3s v1.37.1+k3s1, bundled gateway-api-crd chart, stock klipper-helm v0.13.3-build20260727. The release was already failed at v1. I stopped k3s while the retry job was running and started it again 20s later. End of the retry pod log: The job completed, the release stayed failed at v1, and nothing ran again until I deleted the job about 20 minutes later. Full pod logSo it's the |
|
Yeah that is interesting timing wise - it was able to get the current release status, but then failed to get the actual chart. I am not sure exactly how to handle this; the idea was that helm should create a new release and then the exit 0 allows the job to complete and the helm controller to handle the release status... but in this case it did not even get that far. So clearly we need to handle non-release-generating failures differently. |
|
One way: keep I can send that as a PR to klipper-helm if that's where you want it. |
|
Yeah that would work, as long as doing so doesn't end up being too involved. |
|
Closing in favour of k3s-io/klipper-helm#147, which fixes this in the entry script as discussed. Thanks for the review. |
If a Job completes and the release is still failed at the same revision, the desired Job is identical to the completed one, so apply never replaces it. With
failurePolicy: retrythis happens when the retry'shelm upgradefails before helm stores a release. The chart then stays failed until someone deletes the Job.In that case the controller records the completed Job's UID in a
helmcharts.helm.cattle.io/replacesannotation, so apply replaces the Job, as it does when the config hash annotation changes.The new e2e case fails on master at "Should delete the job to retry the upgrade" and passes with this change.
I used Claude Code to find the cause and write this change and its test.