Skip to content

docs(15.9): restart OpenSearch after restoring configsync so the dictionaries are written - #549

Merged
marevol merged 1 commit into
mainfrom
docs/snapshot-restore-configsync
Sep 23, 2026
Merged

marevol merged 1 commit into
mainfrom
docs/snapshot-restore-configsync

Conversation

@marevol

@marevol marevol commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #542. The snapshot restore procedure restores configsync, calls _configsync/flush, then restores fess*. That only works when the new OpenSearch node was started moments before. On a node that has been up for more than about a minute, the flush writes no dictionary files, fess* restores into a red cluster (IOException while reading mappings_path: file not readable), and the "cluster is red after restore" recovery fails the same way.

Cause

In opensearch-configsync 3.8.0 and earlier (the version in the fess-opensearch:3.8.0 image), _configsync/flush and the scheduled updater share one per-node watermark. Each run searches for files with @timestamp at or after the watermark and moves the watermark to its own start time. Restored files keep the timestamps from the source cluster, so after the first scheduled run (about a minute after startup) neither a flush nor the updater selects them. The file is written only when missing or older than its @timestamp, which does not matter here because it is never selected.

A restart resets the watermark, so a flush right after it writes every stored file. opensearch-configsync#18 (on main, not in a release yet) makes a flush read every stored file, which also fixes this; the restart is harmless there.

Changes

<lang>/15.9/config/admin-index-backup.rst, all seven languages:

  • "Restoring all indexes" step 2: restart OpenSearch (every node), wait for the configsync index, then _configsync/flush. A note explains why the restart is needed and that each written file is logged as Updated <path> by ConfigSyncService.
  • "Cluster is red after restore" step 1: says the restart is part of steps 1 and 2. The wait uses _cluster/health/configsync, which returns while the rest of the cluster is still red.

Verification

Single-node fess-opensearch:3.8.0, a snapshot of configsync plus an index whose analyzer reads two dictionary files through mappings_path and stopwords_path, restored on a fresh node about 95 seconds after it started:

Procedure Files written After restoring the index
restore configsync, flush (current docs) 0 (still 0 after 70 s more) red, 1 unassigned shard
restore configsync, restart, wait, flush (this PR) 2 green, analyzer applies both files
red-cluster recovery: restart, flush, close, restore again 2 green

The same plugin built from opensearch-configsync main wrote both files with flush alone and with restart + flush.

Multi-node clusters were not tested; the restart applies to every node because the watermark is kept per node.

…ionaries are written

The snapshot restore procedure restored the configsync index and then
called _configsync/flush to write the dictionary files. That works only
on an OpenSearch node that was started moments before. With configsync
3.8.0 and earlier, a flush and the scheduled updater share one
per-node watermark: each run writes only files whose @timestamp is at or
after the start of the previous run, and moves the watermark to its own
start time. Restored files keep the timestamps from the cluster the
snapshot was taken on, so once the first scheduled run has completed,
about a minute after startup, the flush writes nothing, fess* restores
into a red cluster, and the documented recovery for a red cluster fails
the same way.

A restart resets the watermark, so the procedure now restarts OpenSearch
(every node) after restoring configsync, waits for the configsync index,
and then flushes. The note explains why the restart is needed and which
log line shows each file being written. The red-cluster recovery refers
to the same steps.

Verified with a single-node fess-opensearch 3.8.0 image and a snapshot
of configsync and an index whose analyzer reads two dictionary files,
restoring about 95 seconds after the new node started:
- restore configsync + flush: 0 files written, index restore red
- restore configsync + restart + flush: 2 files written, index restore
  green, analyzer applies both dictionaries
- red cluster recovery with restart + flush, close, restore: green
@marevol marevol self-assigned this Sep 23, 2026
@marevol
marevol merged commit 71cd0b8 into main Sep 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant