Repository navigation
Rebuild poolmap state changes and improvements - #19335
Draft
wangshilong wants to merge 6 commits into
Draft
wangshilong wants to merge 6 commits into
wangshilong wants to merge 6 commits into
Conversation
co_fseq is the remap order key of the placement. Excluding a target that is being reintegrated from DOWNOUT (DOWNOUT->UP), e.g. by SWIM, used to turn it into DOWN with a new co_fseq, and the failed reintegration was reverted to DOWNOUT later, so the DOWNOUT target ended up with a different co_fseq, which changed the layouts of all objects that have shards remapped from it. Pool map: - MAP_EXCLUDE of a DOWNOUT->UP target only aborts the reintegration, as its data is still on the spare targets: revert it to DOWNOUT with its co_fseq kept, the same as MAP_REVERT_REBUILD, and record the version in co_out_ver. The rank of a SWIM exclusion is reverted to DOWNOUT as well. This also applies to a creation-time DOWNOUT target (co_fseq == co_ver), it stays a creation-time DOWNOUT. DOWN2UP targets still go DOWN with a new co_fseq. - MAP_REVERT_REBUILD of a creation-time DOWNOUT reintegration records the revert version in co_out_ver, the same as the other reverts to DOWNOUT, so pool_map_tgts_dropped_after() detects it. Rebuild: - Add pool_map_tgts_changed_after() to check whether the status of any target of a rebuild task has been changed after the task version. - Add pool_map_tgts_dropped_after() to check whether any target of the pool has been reverted to DOWNOUT after the task version. A rebuild may migrate data to any UP target of its POST_REBUILD layouts, e.g. a reintegrating target. If it is excluded, the shards migrated to it are not rebuilt by its exclusion. pool_map_rebuild_stale_ver() combines both checks. A normal DOWN/DRAIN->DOWNOUT finish is also matched, which only bumps the version of a queued task. - MAP_FINISH_REBUILD of a stale task is rejected with -DER_STALE within the PS RDB TX, so a stale rebuild never marks targets UPIN/DOWNOUT. - The rebuild leader aborts a stale task with -DER_STALE, it is cleaned up by Fail_reclaim and retried, and the retry bumps its version to the version of the change. - A new reintegration of a target is rejected with -DER_BUSY while a running rebuild task is stale for it. - Reintegrating a DOWN target while its exclusion rebuild is running now aborts and retries that rebuild with the new version, instead of letting it finish and mark the reintegrating target UPIN before its data is reintegrated. Add placement tests for the co_fseq/layout of excluding reintegrating (including creation-time DOWNOUT) targets and for the stale detection. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
A rebuild task computes the layouts with its version, while the pool map changes in place. If another target is drained or reintegrated after the task version, the spare that the task is migrating a shard to may be dropped from the write layout and miss the updates, while it holds the shard again (readable) once the task turns the DOWN target into DOWNOUT, so stale data is read. The leader status check only checks new rank level DOWN/REINT and only before the global done, and the finish check only checks the targets of the task and the dropped ones, so this is missed, e.g.: exclude A, rebuild of A migrates to spare B, drain B, finish exclude A, rebuild of A global done, reint C, finish This is a pre-existing issue. pool_map_tgts_dropped_after() now also returns the version of a DRAIN target (co_fseq) or an UP target (co_in_ver) changed after the version, so the rebuild is aborted or its finish is rejected with -DER_STALE, and it is retried with the new version. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
Excluding a draining target was ignored, so the failure was lost: the rank stayed UPIN, the drain might hang on the dead rank, or a failed drain reverted the dead target to UPIN without any rebuild. - MAP_EXCLUDE of a DRAIN target turns it into DOWN with a new co_fseq, the same as an UPIN target, and its data is rebuilt from the other replicas. The drain becomes stale, its Fail_reclaim deletes the data drained to the spares, and the rebuild of the failure may select other spares then. The new co_fseq remaps it after the components changed after the drain start, which is safe as their rebuilds are queued after the drain and nothing computed with the old order has been finished. - The RF check needs no special case: a draining target is still an active, dual-written shard, so it is not a failure, and once it fails it is a new failure since its new co_fseq. - A SWIM exclusion of a draining rank now changes the pool map and goes through the pool RF check like any other rank failure, so it may be rejected with -DER_RF. - Domain DOWN2UP flag is OR-ed instead of overwritten, and DOWN2UP and DOWN2OUT are cleared on UPIN, the same as targets. Add placement tests for the layout, rebuild and the RF check of excluded draining targets. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
MAP_REVERT_REBUILD of a failed DOWN->UP reintegration now gives it a new co_fseq (the same as MAP_EXCLUDE of it), instead of the old one. While it was UP, other rebuilds may have migrated data to it, and the DOWN targets failed after its old co_fseq may have been rebuilt and turned into DOWNOUT, then the old co_fseq hit the DOWN-remapped-through-DOWNOUT assertion of the placement, or skipped it as a spare, losing the data migrated to it. The RF check also counts it as a new failure since the revert then, as the data written to it while UP is lost. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
Excluding a target being extended (NEW->UP, co_fseq < co_ver, which placement skips) gave it a new co_fseq, so it joined the layouts of all existing objects, whose data was never moved to it. - MAP_EXCLUDE of an extending target (and its rank) turns it into DOWN with its co_fseq kept. co_fseq < co_ver of a non-UPIN component means it never joined any layout (pool_comp_is_unplaced()): placement skips such DOWN/DOWNOUT components as it does for NEW, so the layouts are not changed and the exclusion rebuild has nothing to do. A later reintegration of it is a new extension, and a failed one is reverted to DOWN/DOWNOUT instead of NEW. MAP_EXCLUDE of a NEW target is now skipped instead of failing with -DER_NOSYS. - An unplaced DOWN/DOWNOUT target never held data, it is not a failure for the RF check. - pool_map_tgts_dropped_after() also returns an unplaced DOWN target, a rank failure during an extension may rebuild shards to the extending rank, which are not rebuilt by its exclusion. Add placement tests for excluding extending targets/ranks. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
chk_engine marked a dangling target/rank DOWN/DOWNOUT with a new co_fseq directly. Add pool_map_comp_lost_status() and pool_map_comp_mark_lost() to follow the same rules as MAP_EXCLUDE, and keep a dangling NEW target/rank as NEW instead of DOWNOUT, as it is not used by placement. A lost rank also changes its ancestor domains whose children are all DOWN/DOWNOUT, as MAP_EXCLUDE does, otherwise e.g. the node domain added by an extension stays UP and joins the layouts while its only rank is skipped. Add placement tests for the checker marking lost targets the same as MAP_EXCLUDE. Signed-off-by: Wang Shilong <shilong.wang@hpe.com>
|
Errors are component not formatted correctly,Ticket number prefix incorrect,PR title is malformatted. See https://daosio.atlassian.net/wiki/spaces/DC/pages/11133911069/Commit+Comments,Unable to load ticket data |
wangshilong
force-pushed
the
shilongw/rebuild_pb_fixes
branch
from
October 21, 2026 04:28
615f3ee to
9befb3f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Steps for the author:
After all prior steps are complete: