Skip to content

Replace completed job when the release was not updated - #360

Closed
Aureliolo wants to merge 1 commit into
k3s-io:masterfrom
Aureliolo:replace-completed-job
Closed

Aureliolo wants to merge 1 commit into
k3s-io:masterfrom
Aureliolo:replace-completed-job

Conversation

@Aureliolo

Copy link
Copy Markdown

If a Job completes and the release is still failed at the same revision, the desired Job is identical to the completed one, so apply never replaces it. With failurePolicy: retry this happens when the retry's helm upgrade fails before helm stores a release. The chart then stays failed until someone deletes the Job.

In that case the controller records the completed Job's UID in a helmcharts.helm.cattle.io/replaces annotation, so apply replaces the Job, as it does when the config hash annotation changes.

The new e2e case fails on master at "Should delete the job to retry the upgrade" and passes with this change.

I used Claude Code to find the cause and write this change and its test.

Signed-off-by: Aurelio <19254254+Aureliolo@users.noreply.github.com>
@brandond

brandond commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

If a Job completes and the release is still failed at the same revision, the desired Job is identical to the completed one, so apply never replaces it.

Why should the job be recreated? If the helm command is failing and you're trying to get it to continue running until it succeeds, then you should increase .spec.backOffLimit to the number of times you want the job controller to restart the pod. This is just normal kubernetes Job behavior.

With failurePolicy: retry this happens when the retry's helm upgrade fails before helm stores a release.

How exactly would this happen? Helm creates a new release resource before it even starts upgrading, as a way to lock the chart. If that fails then the helm command would fail, and the pod should be restarted as controlled by the backOffLimit - NOT by the job being recreated.

I think you're expecting the controller to do more than it should do.

@Aureliolo

Copy link
Copy Markdown
Author

Hi @brandond, thanks for taking the time to look at this and explain how it's meant to work. Yeah, I think I opened this one too early, sorry.

You're right that helm creates the release first. The case I hit is helm failing before it even gets there. The easiest way I found to get it is restarting k3s while a retry runs: the chart download from the apiserver gets connection refused, the retry branch still does exit 0, the job completes and Traefik (or whichever chart it is) stays failed until someone deletes the job. backOffLimit never kicks in because the pod never fails.

Recreating the job like this PR does would loop forever on stuff that always fails though, so that's not great either. Would it make more sense in klipper-helm? In retry, exit 1 when helm didn't store a new revision, so the kubelet backs off like you described. A failed release that did get stored would still exit 0 like in #140.

To be fair I only ran into it because something else broke the release first, so it's not super common. Fine to close this one either way.

@brandond

brandond commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

The chart download from the apiserver gets connection refused, the retry branch still does exit 0, the job completes and Traefik (or whichever chart it is) stays failed until someone deletes the job.

Can you show logs from this? This does sound like an issue in the klipper-helm entry script or perhaps helm cli itself. If there is an error accessing the chart helm (and the script) should exit with a nonzero exit code.

@Aureliolo

Copy link
Copy Markdown
Author

Sure. k3s v1.37.1+k3s1, bundled gateway-api-crd chart, stock klipper-helm v0.13.3-build20260727. The release was already failed at v1. I stopped k3s while the retry job was running and started it again 20s later.

End of the retry pod log:

+ echo 'Upgrading gateway-api-crd from failed state'
+ helm upgrade --take-ownership gateway-api-crd https://10.43.0.1:443/static/charts/gateway-api-crd-1.6.103.tgz ...
Error: Get "https://10.43.0.1:443/static/charts/gateway-api-crd-1.6.103.tgz": dial tcp 10.43.0.1:443: connect: connection refused
+ exit 0

The job completed, the release stayed failed at v1, and nothing ran again until I deleted the job about 20 minutes later.

Full pod log
21:56:13 if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
21:56:13 	echo "KUBERNETES_SERVICE_HOST is using IPv6"
21:56:13 	CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
21:56:13 else
21:56:13 	CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
21:56:13 fi
21:56:13 
21:56:13 set +v -x
21:56:13 + [[ '' == v2 ]]
21:56:13 + shopt -s nullglob
21:56:13 + [[ -f /config/ca-file.pem ]]
21:56:13 + [[ -f /tmp/ca-file.pem ]]
21:56:13 + [[ false == true ]]
21:56:13 + [[ false == true ]]
21:56:13 + [[ -n '' ]]
21:56:13 + [[ -n AF66F069BEFC569F1F87FC9AB570D2C257969D5FE72EBACC0501595F5EEE93B ]]
21:56:13 + LABELS_ARG='--labels helmcharts.helm.cattle.io/configHash=AF66F069BEFC569F1F87FC9AB570D2C257969D5FE72EBACC0501595F5EEE93B'
21:56:13 + helm_content_decode
21:56:13 + set -e
21:56:13 + ENC_CHART_PATH=/chart/gateway-api-crd.tgz.base64
21:56:13 + CHART_PATH=/tmp/gateway-api-crd.tgz
21:56:13 + [[ ! -f /chart/gateway-api-crd.tgz.base64 ]]
21:56:13 + return
21:56:13 + [[ install != delete ]]
21:56:13 + helm_repo_init
21:56:13 + grep -q -e 'https\?://'
21:56:13 chart path is a url, skipping repo update
21:56:13 + echo 'chart path is a url, skipping repo update'
21:56:13 + helm repo remove stable
21:56:14 Error: no repositories configured
21:56:15 + true
21:56:15 + return
21:56:15 + helm_update install --take-ownership
21:56:15 + read -r STATUS REVISION APPLY_METHOD _
21:56:15 ++ helm status --namespace kube-system gateway-api-crd --output json
21:56:15 ++ jq -r '[.info.status,.version,.apply_method]|@tsv'
21:56:37 ++ get_ssa_args 1 ssa auto true
21:56:37 ++ set -e
21:56:37 ++ REVISION=1
21:56:37 ++ APPLY_METHOD=ssa
21:56:37 ++ SERVER_SIDE=auto
21:56:37 ++ FORCE_CONFLICTS=true
21:56:37 ++ [[ -z 1 ]]
21:56:37 ++ [[ true =~ ^true$ ]]
21:56:37 ++ [[ auto == ^false$ ]]
21:56:37 ++ [[ auto =~ ^auto$ ]]
21:56:37 ++ [[ ! ssa =~ ^ssa$ ]]
21:56:37 ++ [[ -n auto ]]
21:56:37 ++ echo -n '--server-side=auto '
21:56:37 ++ [[ -n true ]]
21:56:37 ++ echo -n '--force-conflicts=true '
21:56:37 ++ echo
21:56:37 ++ set +e
21:56:37 + SSA_ARGS='--server-side=auto --force-conflicts=true '
21:56:37 + VALUES=
21:56:37 + [[ install = delete ]]
21:56:37 + [[ -z 1 ]]
21:56:37 + check_revision 1
21:56:37 + set -e
21:56:37 + REVISION=1
21:56:37 + [[ -n 1 ]]
21:56:37 + [[ 1 != 1 ]]
21:56:37 + set +e
21:56:37 + [[ failed =~ ^(pending-install|pending-upgrade|pending-rollback|superseded|uninstalling)$ ]]
21:56:37 + [[ failed =~ ^deployed$ ]]
21:56:37 + [[ failed =~ ^(deleted|failed|null|unknown)$ ]]
21:56:37 + case "${FAILURE_POLICY:-reinstall}" in
21:56:37 + echo 'Upgrading gateway-api-crd from failed state'
21:56:37 + shift 1
21:56:37 + helm upgrade --take-ownership gateway-api-crd https://10.43.0.1:443/static/charts/gateway-api-crd-1.6.103.tgz --labels helmcharts.helm.cattle.io/configHash=AF66F069BEFC569F1F87FC9AB570D2C257969D5FE72EBACC0501595F5EEE93B --server-side=auto --force-conflicts=true
21:56:37 Error: Get "https://10.43.0.1:443/static/charts/gateway-api-crd-1.6.103.tgz": dial tcp 10.43.0.1:443: connect: connection refused
21:56:37 + exit 0

So it's the exit 0 in the retry branch. Want a PR in klipper-helm that exits 1 there when helm didn't store a new revision?

@brandond

brandond commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Yeah that is interesting timing wise - it was able to get the current release status, but then failed to get the actual chart.

I am not sure exactly how to handle this; the idea was that helm should create a new release and then the exit 0 allows the job to complete and the helm controller to handle the release status... but in this case it did not even get that far. So clearly we need to handle non-release-generating failures differently.

@Aureliolo

Copy link
Copy Markdown
Author

One way: keep exit 0 when helm stored a new revision, since that release carries the failure for the controller. Exit 1 when the revision is still the one read before the upgrade, so the job runs again. As that indicates a general failure or something wrong right?

I can send that as a PR to klipper-helm if that's where you want it.

@brandond

brandond commented Nov 5, 2026

Copy link
Copy Markdown
Member

Yeah that would work, as long as doing so doesn't end up being too involved.

@Aureliolo

Copy link
Copy Markdown
Author

Closing in favour of k3s-io/klipper-helm#147, which fixes this in the entry script as discussed. Thanks for the review.

@Aureliolo Aureliolo closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants