Skip to content

[ISSUE #11521] Reduce timer wheel snapshot flush long-tail - #11522

Open
lizhimins wants to merge 1 commit into
apache:developfrom
lizhimins:timer-segmented-flush
Open

lizhimins wants to merge 1 commit into
apache:developfrom
lizhimins:timer-segmented-flush

Conversation

@lizhimins

Copy link
Copy Markdown
Member

Summary

Fixes #11521.

In timer wheel snapshot mode (timerWheelSnapshotFlush=true), TimerFlushService.flush() held lockWhenFlush across the entire timerlog flush + full-wheel backup() (~36.9 MiB write + force) + checkpoint flush. Since the enqueue path takes the same lock per request, enqueue was blocked for the whole I/O duration, and the single large writeback burst inflated shared-disk CommitLog fsync tail latency. The lock also sat at the call site in fetchAndPutTimerRequest() rather than inside putMessageToTimerWheel(), so a subclass overriding the enqueue loop bypassed it.

This PR:

  • Moves the synchronized (lockWhenFlush) guard inside putMessageToTimerWheel() so every enqueue entry (including subclass overrides) is serialized against the snapshot copy.
  • Rewrites the snapshot branch of flush() to copy-first: freeze the wheel into a reusable buffer under the lock (fast memcpy) and capture wrotePosition as the snapshot flag, then flush the timerlog until it is durable up to the copy's end, write the snapshot, and flush the checkpoint — all outside the lock. A stall guard skips the snapshot if the timerlog flush cannot advance.
  • Writes the snapshot in 1 MiB segments (force(false) per segment, final force(true)) instead of one full-wheel write + force, bounding the writeback burst.
  • Keeps the checkpoint's lastTimerLogFlushPos at the pre-flush position so it never runs ahead of the latest snapshot flag, preserving rollback-to-non-snapshot replay coverage.

The on-disk format is unchanged. The snapshot flag is now the wrotePosition at copy time (previously the post-flush flushedWhere); both are valid 52-byte record boundaries, so recovery via timerlog replay from the max snapshot flag is unchanged and old/new snapshots remain mutually readable. The non-snapshot (default) path is untouched.

Test plan

  • Existing timer unit tests pass (TimerWheelTest, TimerMessageStoreTest, TimerCheckPointTest, TimerLogTest, TimerMetricsTest, TimerEngineSwitchVerifyTest).
  • New TimerWheelTest cases: snapshot copy isolation (live writes after the copy do not leak into the snapshot) and a multi-MiB segmented backup round-trip (per-slot equality across segments).
  • New TimerMessageStoreTest cases: snapshot flush + recovery with concurrent enqueue and long-delay messages (snapshot covers all records, checkpoint stays at/below the flag, offsets/read-time restored after restart, long-delay slots intact); snapshot-mode roll message delivery.
  • Micro-benchmark on a 1C/4Gi broker with a PL0 cloud disk, snapshot mode, 1500 TPS normal + 1000 TPS timer load: send RT p999 reduced ~40-50% versus the previous full-wheel-under-lock backup, across two rebuild-isolated rounds; p50 within round-to-round noise.

…r wheel snapshot flush

In snapshot mode the flush held lockWhenFlush across the whole timerlog
flush + full-wheel backup + checkpoint, blocking enqueue for the entire
I/O duration; the lock also sat at the call site, so subclass enqueue
overrides bypassed it.

- Move the lock inside putMessageToTimerWheel so every enqueue entry is
  serialized against the snapshot copy.
- In flush(), only copy the wheel under the lock and capture wrotePosition
  as the snapshot flag; flush the timerlog to cover the copy's end, then
  write the snapshot and flush the checkpoint outside the lock.
- Write the snapshot in 1 MiB segments (force per segment) to bound the
  writeback burst.
- Keep the checkpoint at the pre-flush position so rollback replay stays
  covered.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 73.91304% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.86%. Comparing base (164322b) to head (958279e).

Files with missing lines Patch % Lines
...apache/rocketmq/store/timer/TimerMessageStore.java 58.62% 8 Missing and 4 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             develop   #11522      +/-   ##
=============================================
+ Coverage      49.79%   49.86%   +0.07%     
- Complexity     14407    14429      +22     
=============================================
  Files           1391     1391              
  Lines         103381   103414      +33     
  Branches       13529    13535       +6     
=============================================
+ Hits           51477    51567      +90     
+ Misses         45686    45625      -61     
- Partials        6218     6222       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement] Reduce enqueue stall and write long-tail of timer wheel snapshot flush

2 participants