ci: put a ceiling on every job, and on the step that hung - #22
Merged
Merged
Conversation
A pull request sat pending for three and a half hours today. Both jobs were stuck in the same place, "Install build dependencies", a step that normally takes between 75 and 193 seconds. apt-get update queries about thirty repositories, and it stopped answering. Nothing failed, because nothing was allowed to. With no timeout-minutes, GitHub applies its 6 hour default, so a hung network call holds a runner for six hours and the pull request reports "pending" the whole time with no indication why. Six of the eight jobs in this repository had no ceiling at all. The two that did are the ones added most recently, which is the usual shape of this: the habit arrived with the new files and nobody went back for the old ones. Ceilings are set from measured durations, with roughly four times the headroom so a cold cache or a slow runner does not trip them: ci.yml test 5.2 min measured -> 20 ci.yml gui 3.2 min measured -> 20 docs.yml build 0.7 min measured -> 15 docs.yml deploy 0.1 min measured -> 10 release.yml release 11.4 min measured -> 45 (Windows, slowest of the matrix) validate validate 0.1 min measured -> 10 Each apt step also gets its own ten minute ceiling. The job-level limit alone would have caught this, but it would have reported that the job timed out; the step-level one names the step that actually stopped responding, which is the difference between a diagnosis and a shrug.
github-actions
Bot
added
the
area:packaging
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pull request sat pending for three and a half hours today. Both jobs were stuck in the same place:
That step normally takes 75 to 193 seconds.
apt-get updatequeries around thirty repositories and stopped answering.Why it lasted three and a half hours
Nothing failed, because nothing was allowed to. With no
timeout-minutes, GitHub applies its 6 hour default. A hung network call holds a runner for six hours while the pull request reportspendingwith no indication why.Six of the eight jobs here had no ceiling. The two that did are the ones added most recently, which is the usual shape of this problem: the habit arrived with the new files and nobody went back for the old ones.
The ceilings
Set from measured durations, with roughly four times the headroom so a cold cache or a slow runner does not trip them.
testguibuilddeployreleasevalidateAlso a ceiling on the apt steps
Each one gets its own ten minutes. The job-level limit alone would have caught today's hang, but it would have reported that the job timed out. The step-level one names the step that actually stopped responding, which is the difference between a diagnosis and a shrug.
Verified
Every workflow still parses, every trigger is unchanged, and the release matrix still has its four platforms. An audit script confirms 8 of 8 jobs now carry a ceiling.