docs(sc): explain generation fault tolerance, with a support matrix - #3841
Open
terrykong wants to merge 2 commits into
Open
docs(sc): explain generation fault tolerance, with a support matrix#3841terrykong wants to merge 2 commits into
terrykong wants to merge 2 commits into
Conversation
Adds docs/guides/single-controller-fault-tolerance.md. Leads with what is and is not supported -- two transports, one generation backend, and no trainer recovery -- because those limits are the first thing someone enabling this needs, and today they can only be recovered by reading the code. The rest is two pictures and a table: who writes and reads the fleet ledger, and what happens for each combination of shard state and refit outcome. The grid is the part that is hard to hold in your head, because the outcome turns on when the shard was noticed rather than on how badly it failed. Calls out the two things that surprise people: is_alive() is answered by the Ray actor and cannot see a wedged engine, and SUSPECT still takes part in a refit. Both are deliberate, and both explain rows in the grid. Signed-off-by: Terry Kong <terryk@nvidia.com>
github-actions
Bot
added
the
Documentation
Three changes from review feedback. Moved to design-docs/, which is where the rest of the refit and generation internals live. Replaced the two mermaid blocks with rendered PNGs. Mermaid laid the ledger out in a way that did not show what was actually being said -- which boxes are states, which are callers, and which direction the arrows mean. The figures are drawn to the same light palette the docs use, and the state list gained RETIRED, which the mermaid version had left out along with its half of absent_shards(). Added an outcome marker to every row of the failure table, so the shape of the answer is visible before the reasoning is read: continues, recovers, depends on transport, or the run ends. Signed-off-by: Terry Kong <terryk@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a guide for the generation fault-tolerance feature this stack builds. Based on part 4 so it can describe the finished thing, restart and re-admission included.
It is deliberately short. Two pictures and a table do most of the work.
What is in it
is_alive()is answered by the Ray actor so it cannot see a wedged engine, andSUSPECTstill takes part in a refit.nccl_reshard, and why.refit_timeout_sis what makes recovery possible at all when a shard fails during a refit.Checked against this branch rather than written from memory: every default in the config block, the two
frozensets that define the serving and absent state sets, which synchronizers overridereconcile_communicator, and which generation backends implementattach_fleet_health.docs/index.mdis updated in both places — the card grid and the toctree.Happy to move it under
design-docs/if you would rather it sat there, or to cut it further.