docs(15.9): restart OpenSearch after restoring configsync so the dictionaries are written - #549
Merged
Merged
Conversation
…ionaries are written The snapshot restore procedure restored the configsync index and then called _configsync/flush to write the dictionary files. That works only on an OpenSearch node that was started moments before. With configsync 3.8.0 and earlier, a flush and the scheduled updater share one per-node watermark: each run writes only files whose @timestamp is at or after the start of the previous run, and moves the watermark to its own start time. Restored files keep the timestamps from the cluster the snapshot was taken on, so once the first scheduled run has completed, about a minute after startup, the flush writes nothing, fess* restores into a red cluster, and the documented recovery for a red cluster fails the same way. A restart resets the watermark, so the procedure now restarts OpenSearch (every node) after restoring configsync, waits for the configsync index, and then flushes. The note explains why the restart is needed and which log line shows each file being written. The red-cluster recovery refers to the same steps. Verified with a single-node fess-opensearch 3.8.0 image and a snapshot of configsync and an index whose analyzer reads two dictionary files, restoring about 95 seconds after the new node started: - restore configsync + flush: 0 files written, index restore red - restore configsync + restart + flush: 2 files written, index restore green, analyzer applies both dictionaries - red cluster recovery with restart + flush, close, restore: green
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #542. The snapshot restore procedure restores
configsync, calls_configsync/flush, then restoresfess*. That only works when the new OpenSearch node was started moments before. On a node that has been up for more than about a minute, the flush writes no dictionary files,fess*restores into a red cluster (IOException while reading mappings_path: file not readable), and the "cluster is red after restore" recovery fails the same way.Cause
In opensearch-configsync 3.8.0 and earlier (the version in the
fess-opensearch:3.8.0image),_configsync/flushand the scheduled updater share one per-node watermark. Each run searches for files with@timestampat or after the watermark and moves the watermark to its own start time. Restored files keep the timestamps from the source cluster, so after the first scheduled run (about a minute after startup) neither a flush nor the updater selects them. The file is written only when missing or older than its@timestamp, which does not matter here because it is never selected.A restart resets the watermark, so a flush right after it writes every stored file. opensearch-configsync#18 (on
main, not in a release yet) makes a flush read every stored file, which also fixes this; the restart is harmless there.Changes
<lang>/15.9/config/admin-index-backup.rst, all seven languages:configsyncindex, then_configsync/flush. A note explains why the restart is needed and that each written file is logged asUpdated <path>byConfigSyncService._cluster/health/configsync, which returns while the rest of the cluster is still red.Verification
Single-node
fess-opensearch:3.8.0, a snapshot ofconfigsyncplus an index whose analyzer reads two dictionary files throughmappings_pathandstopwords_path, restored on a fresh node about 95 seconds after it started:The same plugin built from opensearch-configsync
mainwrote both files with flush alone and with restart + flush.Multi-node clusters were not tested; the restart applies to every node because the watermark is kept per node.