Skip to content

DAOS-19591 container: recover EC boundaries after reintegration (#19321) - #19350

Merged
mchaarawi merged 1 commit into
release/2.6.5-patchfrom
mschaara/19591-2.6.5
Oct 23, 2026
Merged

mchaarawi merged 1 commit into
release/2.6.5-patchfrom
mschaara/19591-2.6.5

Conversation

@mchaarawi

Copy link
Copy Markdown
Contributor

Reintegrated or extended targets can miss the last EC aggregation boundary and wait indefinitely for a refresh. Their zero epochs may be suppressed by the rank's previous report, or they may have no local VOS container and therefore never report. Refreshing only when the minimum epoch changes can leave the container's boundary and space reclamation stuck.

Track such waiting ranks with ea_need_refresh, set when a rank's zero report is suppressed by its earlier report or when some non-failed rank has reported nothing yet. While it is set, and rebuild is inactive and reclaim is enabled, resend the unchanged boundary every 30 seconds; the flag is cleared before each push so steady-state traffic drops back to zero. Allow leader pushes to create missing VOS containers, using deep-stack thread collectives and tolerating concurrent creation. Mark pushes using the otherwise unused boundary payload rank; clear that marker on fetch copies so cached boundaries cannot recreate containers. The IV wire layout remains unchanged.

When destroying a container, wait up to 10 seconds for an in-flight boundary push and return DER_BUSY if it has not finished, rather than proceeding after a timeout. Stop tracking before the destroy broadcast, including orphan cleanup, and check retired tracking entries as well. Skip refreshes after RDB errors or deletion during an RDB operation, and retire entries for nonexistent containers.

Also suppress sluggish-boundary warnings for epoch jumps larger than the current leader's tracking age.

When a target is excluded and reintegrated on a running engine (for example an evicted SSD), its container shards start with no EC aggregation boundary and wait for the leader to resend it. The rank's reporting ULT however skipped a zero minimum once it had reported a non-zero epoch before, so the leader never learned that a shard was waiting, kept the rank's stale epoch and pinned the pool-wide boundary.

Report a zero minimum even after earlier non-zero reports. The leader ignores the value itself but marks the container for a boundary resend, which validates the boundary on the reintegrated target and lets it aggregate and report again.

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

Reintegrated or extended targets can miss the last EC aggregation
boundary and wait indefinitely for a refresh. Their zero epochs may be
suppressed by the rank's previous report, or they may have no local VOS
container and therefore never report. Refreshing only when the minimum
epoch changes can leave the container's boundary and space reclamation
stuck.

Track such waiting ranks with ea_need_refresh, set when a rank's zero
report is suppressed by its earlier report or when some non-failed rank
has reported nothing yet. While it is set, and rebuild is inactive and
reclaim is enabled, resend the unchanged boundary every 30 seconds; the
flag is cleared before each push so steady-state traffic drops back to
zero. Allow leader pushes to create missing VOS containers, using
deep-stack thread collectives and tolerating concurrent creation. Mark
pushes using the otherwise unused boundary payload rank; clear that
marker on fetch copies so cached boundaries cannot recreate containers.
The IV wire layout remains unchanged.

When destroying a container, wait up to 10 seconds for an in-flight
boundary push and return DER_BUSY if it has not finished, rather than
proceeding after a timeout. Stop tracking before the destroy broadcast,
including orphan cleanup, and check retired tracking entries as well.
Skip refreshes after RDB errors or deletion during an RDB operation, and
retire entries for nonexistent containers.

Also suppress sluggish-boundary warnings for epoch jumps larger than
the current leader's tracking age.

When a target is excluded and reintegrated on a running engine (for
example an evicted SSD), its container shards start with no EC
aggregation boundary and wait for the leader to resend it. The rank's
reporting ULT however skipped a zero minimum once it had reported a
non-zero epoch before, so the leader never learned that a shard was
waiting, kept the rank's stale epoch and pinned the pool-wide boundary.

Report a zero minimum even after earlier non-zero reports. The leader
ignores the value itself but marks the container for a boundary resend,
which validates the boundary on the reintegrated target and lets it
aggregate and report again.

Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
@github-actions

Copy link
Copy Markdown

Ticket title is 'Aurora daos_user: engine printing continuously in log "cont_agg_eph_sync() 66e9ae82/2416cc43: Sluggish EC boundary report from rank 117, 165933 Seconds." '
Status is 'In Review'
Labels: 'ALCF'
https://daosio.atlassian.net/browse/DAOS-19591

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19350/1/testReport/

@liuxuezhao

Copy link
Copy Markdown
Contributor

test failures are NLT, and dfuse/daos_build.py, looks unrelated with this pR

@mchaarawi
mchaarawi marked this pull request as ready for review October 23, 2026 12:28
@mchaarawi
mchaarawi requested review from a team as code owners October 23, 2026 12:28
@mchaarawi
mchaarawi merged commit 188a4e8 into release/2.6.5-patch Oct 23, 2026
41 of 45 checks passed
@mchaarawi
mchaarawi deleted the mschaara/19591-2.6.5 branch October 23, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants