test: feature_llmq_signing.py --spork21 intermittent timeout in wait_for_sigs

Summary

feature_llmq_signing.py --spork21 intermittently fails with a wait_until() timeout in wait_for_sigs(). The failure is timing-dependent and not reproducible — the same test passes on develop and the non---spork21 variant passes consistently on the same CI run.

Failure Details

Test: feature_llmq_signing.py --spork21
Error: AssertionError: Predicate '' not true after N seconds
Location: feature_llmq_signing.py:111 → wait_for_sigs(True, False, True, 15) (15s timeout)
Traceback:

File "feature_llmq_signing.py", line 111, in run_test
    wait_for_sigs(True, False, True, 15)
File "feature_llmq_signing.py", line 60, in wait_for_sigs
    self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)
AssertionError: Predicate '' not true after <timeout> seconds

Retry behavior: Failed all 3 CI retry attempts (attempt 1: 128s, attempt 2: 114s, attempt 3: 131s)

CI Log Links

Failing run on PR fix: register shareman recovery interface in ActiveContext #7244: https://github.com/dashpay/dash/actions/runs/23441584574/job/68207659502
Passing run on develop (same commit after merge): https://github.com/dashpay/dash/actions/runs/23505322590 — linux64_multiprocess-test / Test source passed

Evidence of Flakiness

The non---spork21 variant (feature_llmq_signing.py) passed on the exact same CI run
The --spork21 variant passed on develop after the PR was merged (CI run 23505322590)
PR fix: register shareman recovery interface in ActiveContext #7244 only modifies src/active/context.cpp (adds shareman->RegisterRecoveryInterface()) — no test files or LLMQ signing logic changed

Root Cause Analysis

The 15-second timeout in wait_for_sigs() appears insufficient for the multiprocess test environment under CI load. The --spork21 variant likely exercises a slightly different code path that takes marginally longer, hitting the timeout under resource contention.

Suggested Fix

Consider increasing the timeout in wait_for_sigs() calls within the --spork21 path, or using self.options.timeout_factor to scale timeouts appropriately for CI environments.

Another occurrence of this flaky test failure:

CI run: https://github.com/dashpay/dash/actions/runs/23780028947/job/69294959989
PR: #7251 (ci: convert clang-format diff check to informational PR comment — only changes .github/workflows/clang-diff-format.yml)
Build target: linux64_multiprocess
Date: 2026-03-31

The test failed 3/3 attempts with the same wait_until() timeout:

AssertionError: Predicate '' not true after N seconds

Other LLMQ tests also failed on first attempt during this run (feature_llmq_data_recovery, feature_llmq_simplepose, feature_llmq_dkgerrors) but passed on retry — suggesting general LLMQ test instability on ARM runners.

Another occurrence from PR #7242 (wallet deprecation + RPC type checking backport), but this one is unrelated to the PR changes.

CI run: https://github.com/dashpay/dash/actions/runs/24064815059/job/70192274452
Build target: linux64_multiprocess
Date: 2026-04-07

The failure was again feature_llmq_signing.py --spork21, timing out in wait_for_sigs(True, False, True, 15) after retrying 3/3 times:

AssertionError: Predicate "" not true after 60.0 seconds

Same run also had separate in-scope addressindex RPC type-check failures in other jobs; those were fixed on the PR branch independently. This multiprocess failure still looks like the existing intermittent LLMQ signing flake.

Another occurrence from PR #7232, again unrelated to the PR diff itself (the PR only changes CI/workflow/env files plus test/get_previous_releases.py).

CI run: https://github.com/dashpay/dash/actions/runs/24081094106
Job: linux64-test / Test source (70248433408)
Date: 2026-04-07

This run hit the same feature_llmq_signing.py DKG phase timeout in wait_for_quorum_phase():

AssertionError: Predicate ... not true after 120.0 seconds

Same job also had separate collateral failures from a missing shared RPC coverage temp dir (dashpay/dash#7273), but the LLMQ signing timeout is still the existing intermittent failure tracked here.

Another occurrence from PR #7242 (wallet deprecation + RPC type checking backport), current head 24daadcf6d, and it is still unrelated to the PR diff itself.

CI run: https://github.com/dashpay/dash/actions/runs/24580460650
Job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71879977938
Build target: linux64_sqlite-test / Test source
Date: 2026-04-17

Failure was again feature_llmq_signing.py --spork21, timing out in wait_for_sigs(True, False, True, 15) after retrying 3/3 times:

AssertionError: Predicate "" not true after 60.0 seconds

I rechecked the current PR head and git log upstream/develop..HEAD -- test/functional/feature_llmq_signing.py is empty, so this test file is still untouched by the branch. The only functional-test changes on the PR are unrelated RPC bool/type-check adaptations plus deprecatedrpc=create_bdb test config plumbing. This still looks like the existing intermittent LLMQ signing flake tracked here, not a regression from #7242.

Another occurrence from dashpay/dash#7232 after the multiprocess lane had already been moved back to amd64:

PR head: ff5ff3536502914506ce5c912783a1e070f8a7f1
Failing job: linux64_multiprocess-test / Test source
Run/job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
Failing variant: feature_llmq_signing.py --spork21
Failure mode: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

This is useful because the latest failure is no longer tied to the PR's ARM-runner migration path for multiprocess: on this head, linux64_multiprocess depends/build/test all run on amd64 again, yet the same intermittent timeout still reproduces in CI.

I also re-ran the exact failing variant locally on upstream/develop:

python3 test/functional/feature_llmq_signing.py --spork21

That local develop run passed, which fits the existing intermittent/timing-sensitive diagnosis rather than a deterministic regression from dash#7232.

Another confirmation from tracker #1015 / #7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
failing variant: feature_llmq_signing.py --spork21
failure mode: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

I rechecked the current branch diff and it still does not touch test/functional/feature_llmq_signing.py (git log upstream/develop..HEAD -- test/functional/feature_llmq_signing.py is empty). I also confirmed the latest linux64_multiprocess-test lane on this PR is already running on amd64 again, so this occurrence is not specific to the ARM-runner migration path anymore.

I reran the exact failing variant locally on upstream/develop again:

python3 test/functional/feature_llmq_signing.py --spork21

That local develop run passed, which is consistent with the existing intermittent/timing-sensitive flake diagnosis rather than a deterministic regression from #7232.

Another datapoint from tracker #1023 / PR #7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
failing variant: feature_llmq_signing.py --spork21
branch diff still does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/ (git log upstream/develop..HEAD -- <path> is empty for both)

I also reran the exact failing variant locally on upstream/develop just now:

python3 test/functional/feature_llmq_signing.py --spork21

This time it failed locally on develop with the same timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111:

AssertionError: Predicate """"
            self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)
""" not true after 15 seconds

So the latest #7232 occurrence is not only unrelated to the PR diff, it is now directly reproducible on develop as the same underlying test bug.

Another recheck from tracker #1029 / PR #7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
failing variant: feature_llmq_signing.py --spork21
branch diff still only touches CI/workflow/env files plus test/get_previous_releases.py; it does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/
the current linux64_multiprocess-test lane on this PR head is already back on amd64, so this is still not specific to the ARM-runner migration path

I reran the exact failing variant locally on upstream/develop again today:

python3 test/functional/feature_llmq_signing.py --spork21

That rerun passed on develop, which keeps matching the intermittent/timing-sensitive flake diagnosis rather than a deterministic regression from dash#7232. Existing comments on this issue already show mixed local pass/fail outcomes for the same test, which strengthens the flake diagnosis.

@thepastaclaw

@thepastaclaw feature_llmq_signing.py --spork21 has intermittent failure; it may fail or not depends to some multi-thread race or miscommunications between nodes / communications with wrong unexpected timing.

Single run of test python3 test/functional/feature_llmq_signing.py --spork21 can not be used to determine a successful fix or still having issue and separate a fixed version from the broken.

Do at least 10 runs and validate a result, count amount of failures with --spork21:
test/functional/test_runner.py -j20 feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py

consider reducing -j20 to smaller amount if you have not enough RAM or less than 4 cpu cores and increasing amount of runs from 10 to 20, 50 if results are not clear.

Drop failure rate to the half of current failure rate is a considered as a proper fix worth to get merged.

@knst

@knst agreed — single-run pass/fail isn't a useful validation signal here.

I ran a 10x batch on current upstream/develop (cfad4141d60961e5d4f27e11a8d30deccf18250a) with:

python3 test/functional/test_runner.py -j10 --timeout-factor=1 \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21"

Result: 6/10 failures, 4/10 passes.

Failure breakdown from this batch:

5/6 failures timed out in the same place already tracked here:
- feature_llmq_signing.py:111
- wait_for_sigs(True, False, True, 15)
1/6 failures made it further, then failed at:
- feature_llmq_signing.py:162
- assert_sigs_nochange(True, False, True, 3)

So the current baseline on develop is roughly a 60% failure rate under repeated runs, and it is not limited to a single lucky/unlucky invocation. I'll use repeated-run batches like this for validation going forward, and any candidate fix should reduce that failure rate materially (per your "drop it by at least half" bar).

Another occurrence from thepastaclaw/dash#21 (backport: bitcoin#27594 — refactor: Remove unused GetTimeMillis), and this one is again unrelated to the PR diff itself.

PR head: d0e55ea865ef510106abb597feae88dd4f68507d
failing jobs:
- https://github.com/thepastaclaw/dash/actions/runs/24655377399/job/72093830252 (linux64-test / Test source)
- https://github.com/thepastaclaw/dash/actions/runs/24655377399/job/72093138446 (linux64_sqlite-test / Test source)
- https://github.com/thepastaclaw/dash/actions/runs/24655375875/job/72098129353 (linux64_multiprocess-test / Test source)
failing variant in all three jobs: feature_llmq_signing.py --spork21
failure mode:
- feature_llmq_signing.py:111 / wait_for_sigs(True, False, True, 15) in linux64-test and linux64_multiprocess-test
- feature_llmq_signing.py:201 / wait_for_sigs(True, False, True, 2) in linux64_sqlite-test

Scope check on the PR branch:

exact PR diff only touches src/rpc/net.cpp and src/util/time.cpp
git log upstream/develop..HEAD -- test/functional/feature_llmq_signing.py test/functional/test_framework/ is empty on the PR head

Also useful: the same head already has successful runs for those same job names in the other matrix run attached to the PR, so this occurrence again shows pass/fail behavior on an unchanged commit.

I reran the exact failing variant locally on current develop:

python3 test/functional/feature_llmq_signing.py --spork21

That single run passed for me today, which is still consistent with the existing intermittent diagnosis rather than a deterministic regression from this PR.

Another occurrence from tracker #1035 / dashpay/dash#7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
failing variant: feature_llmq_signing.py --spork21
failure mode: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

I rechecked the PR diff on this head and it still only touches CI/workflow/env files plus test/get_previous_releases.py; it does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/ (git log upstream/develop..HEAD -- <path> is empty for both).

I also reran the exact failing variant locally on current upstream/develop just now:

python3 test/functional/feature_llmq_signing.py --spork21

That run passed for me this time. Together with the existing mixed pass/fail data already on this issue, this still points to the same intermittent/timing-sensitive flake rather than a deterministic regression from dash#7232.

Another occurrence on #7242 (head 24daadcf6d, run https://github.com/dashpay/dash/actions/runs/24580460650):

failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71879977938
failure: feature_llmq_signing.py --spork21 timed out again in wait_for_sigs(True, False, True, 15) after failing all 3 CI retry attempts

Scope re-check on this PR still points away from branch-caused regression:

git diff upstream/develop..HEAD --name-only does not touch test/functional/feature_llmq_signing.py
the non---spork21 variant passed in the same CI job
I also ran python3 test/functional/feature_llmq_signing.py --spork21 on current develop locally and it passed

So this remains consistent with the intermittent timeout/flakiness already tracked here, not something introduced by PR #7242.

Another occurrence from tracker #1067 / dashpay/dash#7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
failing variant: feature_llmq_signing.py --spork21
failure mode: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

I rechecked the exact PR diff on this head with git diff upstream/develop..HEAD --name-only; it still only touches CI/workflow/env files plus test/get_previous_releases.py, and does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/.

I also reran the exact failing variant locally on current upstream/develop just now:

python3 test/functional/feature_llmq_signing.py --spork21

This time it failed locally on develop with the same timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111:

AssertionError: Predicate ''''
            self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)
''' not true after 15 seconds

That makes this latest #7232 failure directly reproducible on develop again, so it is still the pre-existing intermittent LLMQ signing test bug tracked here, not a regression from the PR branch.

Another occurrence from tracker #1074 / dashpay/dash#7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
failing variant: feature_llmq_signing.py --spork21
failure mode in CI: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

I rechecked the exact PR diff on this head with git diff upstream/develop..HEAD --name-only; it still only touches CI/workflow/env files plus test/get_previous_releases.py, and does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/ (git log upstream/develop..HEAD -- test/functional/feature_llmq_signing.py test/functional/test_framework/ is empty).

I also reran the exact failing variant locally on current upstream/develop just now:

./test/functional/feature_llmq_signing.py --spork21

This time it failed locally on develop too, with the same underlying signing-timeout predicate (different assertion site later in the script, but same check_sigs(...) wait):

File "/Users/claw/Projects/dash/./test/functional/feature_llmq_signing.py", line 201, in run_test
    wait_for_sigs(True, False, True, 2)
...
AssertionError: Predicate '\\\\n            self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)\n'\\ not true after 2 seconds\n```\n\nSo the latest `#7232` failure is again reproducible on plain develop and remains the pre-existing intermittent `feature_llmq_signing.py --spork21` bug tracked here, not a regression from the PR branch.

Another occurrence on PR #7242 (head 24daadcf6dd495c76b5ec2ab6c1e6c9dfec979b8):

failing workflow run: https://github.com/dashpay/dash/actions/runs/24580460650
failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71879977938

Same failure shape as before:

feature_llmq_signing.py --spork21
times out in wait_for_sigs(True, False, True, 15)
traceback still ends in AssertionError: Predicate "" not true after ... seconds

Scope check for PR #7242 still shows this is unrelated to the branch changes: the PR does not touch test/functional/feature_llmq_signing.py, and the only touched functional helper file is test/functional/test_framework/test_framework.py in unrelated RPC bool/type-check adaptation hunks.

Another occurrence from tracker #1093 / dashpay/dash#7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
failing variant: feature_llmq_signing.py --spork21
failure mode in CI: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

Scope re-check on this PR still points away from branch-caused regression:

git diff upstream/develop..HEAD --name-only on the PR head still only touches CI/workflow/env files plus test/get_previous_releases.py
the branch does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/

I also reran the exact failing variant locally on plain upstream/develop just now from /Users/claw/Projects/dash:

./test/functional/feature_llmq_signing.py --spork21

That local develop run failed with the same timeout shape:

File "/Users/claw/Projects/dash/./test/functional/feature_llmq_signing.py", line 111, in run_test
    wait_for_sigs(True, False, True, 15)
...
AssertionError: Predicate ''''
            self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)
''' not true after 15 seconds

So this latest #7232 failure is again directly reproducible on develop and remains the pre-existing intermittent feature_llmq_signing.py --spork21 bug tracked here, not a regression from the PR branch.

Fresh tracker-1107 scope recheck on PR #7242 head 24daadcf6d:\n\n- failing workflow run: https://github.com/dashpay/dash/actions/runs/24580460650\n- failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71879977938\n- failure: feature_llmq_signing.py --spork21 timed out again in wait_for_sigs(True, False, True, 15)\n\nI rechecked the exact branch diff with git diff upstream/develop..HEAD --name-only on the PR head. It still does not touch test/functional/feature_llmq_signing.py. The only functional-helper changes on this branch are unrelated RPC bool/type-check adaptations plus deprecatedrpc=create_bdb config plumbing.\n\nSo this occurrence is still out-of-scope for PR #7242 itself and remains the pre-existing intermittent flake tracked here. No PR-branch mutation is appropriate for this failure alone.

Merge #7289: revert: "fix: avoid improper dual way connection attempt…

Fixed by #7289

Navigation Menu

Repository navigation

test: feature_llmq_signing.py --spork21 intermittent timeout in wait_for_sigs #7255

Summary

Failure Details

CI Log Links

Evidence of Flakiness

Root Cause Analysis

Suggested Fix

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

test: feature_llmq_signing.py --spork21 intermittent timeout in wait_for_sigs #7255

Description

Summary

Failure Details

CI Log Links

Evidence of Flakiness

Root Cause Analysis

Suggested Fix

Activity

thepastaclaw commented on Mar 31, 2026

thepastaclaw commented on Apr 7, 2026

thepastaclaw commented on Apr 7, 2026

thepastaclaw commented on Apr 18, 2026

thepastaclaw commented on Apr 19, 2026

thepastaclaw commented on Apr 19, 2026

thepastaclaw commented on Apr 19, 2026

thepastaclaw commented on Apr 20, 2026

knst commented on Apr 20, 2026

thepastaclaw commented on Apr 20, 2026

thepastaclaw commented on Apr 20, 2026

thepastaclaw commented on Apr 21, 2026

thepastaclaw commented on Apr 21, 2026

thepastaclaw commented on Apr 21, 2026

thepastaclaw commented on Apr 23, 2026

thepastaclaw commented on Apr 24, 2026

thepastaclaw commented on Apr 26, 2026

thepastaclaw commented on Apr 28, 2026

knst commented on May 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions