Repository navigation
DAOS-19591 container: recover EC boundaries after reintegration (#19321) - #19350
Merged
Merged
Conversation
Reintegrated or extended targets can miss the last EC aggregation boundary and wait indefinitely for a refresh. Their zero epochs may be suppressed by the rank's previous report, or they may have no local VOS container and therefore never report. Refreshing only when the minimum epoch changes can leave the container's boundary and space reclamation stuck. Track such waiting ranks with ea_need_refresh, set when a rank's zero report is suppressed by its earlier report or when some non-failed rank has reported nothing yet. While it is set, and rebuild is inactive and reclaim is enabled, resend the unchanged boundary every 30 seconds; the flag is cleared before each push so steady-state traffic drops back to zero. Allow leader pushes to create missing VOS containers, using deep-stack thread collectives and tolerating concurrent creation. Mark pushes using the otherwise unused boundary payload rank; clear that marker on fetch copies so cached boundaries cannot recreate containers. The IV wire layout remains unchanged. When destroying a container, wait up to 10 seconds for an in-flight boundary push and return DER_BUSY if it has not finished, rather than proceeding after a timeout. Stop tracking before the destroy broadcast, including orphan cleanup, and check retired tracking entries as well. Skip refreshes after RDB errors or deletion during an RDB operation, and retire entries for nonexistent containers. Also suppress sluggish-boundary warnings for epoch jumps larger than the current leader's tracking age. When a target is excluded and reintegrated on a running engine (for example an evicted SSD), its container shards start with no EC aggregation boundary and wait for the leader to resend it. The rank's reporting ULT however skipped a zero minimum once it had reported a non-zero epoch before, so the leader never learned that a shard was waiting, kept the rank's stale epoch and pinned the pool-wide boundary. Report a zero minimum even after earlier non-zero reports. The leader ignores the value itself but marks the container for a boundary resend, which validates the boundary on the reintegrated target and lets it aggregate and report again. Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
|
Ticket title is 'Aurora daos_user: engine printing continuously in log "cont_agg_eph_sync() 66e9ae82/2416cc43: Sluggish EC boundary report from rank 117, 165933 Seconds." ' |
Collaborator
|
Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19350/1/testReport/ |
Collaborator
|
Test stage Functional Hardware Medium completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19350/1/testReport/ |
Contributor
|
test failures are NLT, and dfuse/daos_build.py, looks unrelated with this pR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reintegrated or extended targets can miss the last EC aggregation boundary and wait indefinitely for a refresh. Their zero epochs may be suppressed by the rank's previous report, or they may have no local VOS container and therefore never report. Refreshing only when the minimum epoch changes can leave the container's boundary and space reclamation stuck.
Track such waiting ranks with ea_need_refresh, set when a rank's zero report is suppressed by its earlier report or when some non-failed rank has reported nothing yet. While it is set, and rebuild is inactive and reclaim is enabled, resend the unchanged boundary every 30 seconds; the flag is cleared before each push so steady-state traffic drops back to zero. Allow leader pushes to create missing VOS containers, using deep-stack thread collectives and tolerating concurrent creation. Mark pushes using the otherwise unused boundary payload rank; clear that marker on fetch copies so cached boundaries cannot recreate containers. The IV wire layout remains unchanged.
When destroying a container, wait up to 10 seconds for an in-flight boundary push and return DER_BUSY if it has not finished, rather than proceeding after a timeout. Stop tracking before the destroy broadcast, including orphan cleanup, and check retired tracking entries as well. Skip refreshes after RDB errors or deletion during an RDB operation, and retire entries for nonexistent containers.
Also suppress sluggish-boundary warnings for epoch jumps larger than the current leader's tracking age.
When a target is excluded and reintegrated on a running engine (for example an evicted SSD), its container shards start with no EC aggregation boundary and wait for the leader to resend it. The rank's reporting ULT however skipped a zero minimum once it had reported a non-zero epoch before, so the leader never learned that a shard was waiting, kept the rank's stale epoch and pinned the pool-wide boundary.
Report a zero minimum even after earlier non-zero reports. The leader ignores the value itself but marks the container for a boundary resend, which validates the boundary on the reintegrated target and lets it aggregate and report again.
Steps for the author:
After all prior steps are complete: