[ISSUE #10559] Support wildcard pattern matching for LiteTopic subscriptions

[Aias00]

Which issue does this PR close?

Closes #10559.

What's the purpose of this PR

Add NATS-style wildcard pattern matching so a wildcard LiteTopic consumer group can subscribe to a subset of lite-topics under its parent topic (instead of always receiving all of them), using __ as the segment separator. Full backward compatibility is preserved: wildcard groups without patterns behave identically to before.

Brief changelog

• LitePatternMatcher (new, common) — the matching engine: validate, matches (recursive, no regex), matchesAny, expand. * matches exactly one __-delimited segment; ** matches one or more trailing segments (like NATS >).
• LiteSubscription — add in-memory wildcardPatterns set (rebuilt on reconnect, same as liteTopicSet).
• LiteSubscriptionRegistry[Impl] — a wildcard group with non-empty validated patterns enters pattern mode: eagerly expand against collectByParentTopic and register real lmqNames. Empty patterns keep legacy receive-all behavior (synthetic topic@group key) unchanged. getAllSubscriber always merges the synthetic-key clients for any wildcard group, so mixed pattern/legacy groups deliver to both. New reexpandWildcardPatterns picks up lite-topics created after the initial subscription.
• LiteSubscriptionCtlProcessor — route wildcard COMPLETE_ADD through toWildcardPatterns (patterns ride the existing liteTopicSet — no wire protocol change). Non-empty but all-invalid pattern input is rejected instead of silently widening to receive-all.
• LiteEventDispatcher.doFullDispatchForWildcardGroup — replace the O(N) forEachLiteTopic scan with O(M) collectByParentTopic; re-expand patterns for new LMQs and clear backlog per pattern-mode client.

Pattern semantics

Pattern	Matches	Does NOT match
pay__refund	pay__refund	pay__refund__notify, notify__refund
pay__*	pay__refund, pay__success	pay__refund__notify
*__refund	pay__refund, notify__refund	pay__refund__notify
pay__*__notify	pay__refund__notify, pay__success__notify	pay__refund
**	all	—
pay__**	pay__refund, pay__refund__notify	notify__refund

How to test / verify

• New LitePatternMatcherTest (common): all six issue rows (match + non-match), validate accept/reject cases, expand/matchesAny semantics — 18 tests.
• Updated registry/dispatcher/processor tests, split into legacy + pattern-mode variants; added mixed-group delivery, all-invalid-pattern rejection, and lazy re-expansion coverage.
• mvn -pl common,broker -am test — lite suite green (138 tests, 0 failures); checkstyle clean on common + broker.
• Performance sanity: matches ≈ 1µs/call (target <5µs); expand(1k candidates × 5 patterns) ≈ 360µs (target <10ms).

Scope

Broker + common only, matching the issue's "Key Changes" table. Patterns are carried via the existing LiteSubscriptionDTO.liteTopicSet; the broker distinguishes them via isWildcardGroup(). A consumer/proxy API (subscribeLite) and a true LMQ-creation listener for sub-second new-topic pickup are intentionally deferred to follow-up PRs.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds NATS-style wildcard pattern matching for LiteTopic wildcard consumer groups, enabling clients to subscribe to a subset of lite-topics under a parent topic (using __-delimited segments) while preserving legacy “receive all” behavior when no patterns are provided. It also optimizes wildcard-group full dispatch by iterating only lite-topics under the bound parent topic.

Changes:

• Introduces LitePatternMatcher (common) with validate, matches, matchesAny, and expand, plus comprehensive unit tests.
• Extends lite subscription/registry flow to support “pattern mode” wildcard groups by storing patterns and eagerly/lazily expanding them to concrete LMQ names.
• Updates broker control/dispatch paths and tests to route wildcard COMPLETE_ADD through pattern handling and to avoid full LMQ table scans.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
common/src/main/java/org/apache/rocketmq/common/lite/LitePatternMatcher.java	Adds wildcard pattern validation/matching/expansion for lite-topic child names.
common/src/test/java/org/apache/rocketmq/common/lite/LitePatternMatcherTest.java	Tests pattern semantics, validation rules, and expansion behavior.
common/src/main/java/org/apache/rocketmq/common/lite/LiteSubscription.java	Adds in-memory wildcardPatterns storage for pattern-mode wildcard groups.
broker/src/main/java/org/apache/rocketmq/broker/processor/LiteSubscriptionCtlProcessor.java	Routes wildcard COMPLETE_ADD to registry overload carrying validated patterns.
broker/src/test/java/org/apache/rocketmq/broker/processor/LiteSubscriptionCtlProcessorTest.java	Adds tests for wildcard routing and all-invalid-pattern rejection.
broker/src/main/java/org/apache/rocketmq/broker/lite/LiteSubscriptionRegistry.java	Adds registry overload + reexpandWildcardPatterns API for pattern-mode support.
broker/src/main/java/org/apache/rocketmq/broker/lite/LiteSubscriptionRegistryImpl.java	Implements pattern-mode expansion, mixed legacy+pattern delivery merging, and re-expansion.
broker/src/test/java/org/apache/rocketmq/broker/lite/LiteSubscriptionRegistryImplTest.java	Adds tests for pattern-mode expansion/resubscribe, re-expansion, and mixed-group delivery.
broker/src/main/java/org/apache/rocketmq/broker/lite/LiteEventDispatcher.java	Optimizes wildcard full dispatch via collectByParentTopic and adds pattern-client re-expansion.
broker/src/test/java/org/apache/rocketmq/broker/lite/LiteEventDispatcherTest.java	Adds tests covering legacy vs pattern-mode wildcard full dispatch paths.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 81.22271% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.25%. Comparing base (0e4ccf1) to head (2c59764).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
...etmq/broker/lite/LiteSubscriptionRegistryImpl.java	79.27%	8 Missing and 15 partials ⚠️
.../apache/rocketmq/common/lite/LiteSubscription.java	0.00%	7 Missing ⚠️
...pache/rocketmq/common/lite/LitePatternMatcher.java	89.65%	2 Missing and 4 partials ⚠️
...ache/rocketmq/broker/lite/LiteEventDispatcher.java	83.87%	3 Missing and 2 partials ⚠️
...broker/processor/LiteSubscriptionCtlProcessor.java	90.90%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop   #10607      +/-   ##
=============================================
- Coverage      48.27%   48.25%   -0.02%     
- Complexity     13436    13503      +67     
=============================================
  Files           1378     1379       +1     
  Lines         100820   101027     +207     
  Branches       13041    13105      +64     
=============================================
+ Hits           48670    48755      +85     
- Misses         46208    46288      +80     
- Partials        5942     5984      +42     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[RockteMQ-AI]

Review by github-manager-bot

Summary

This PR adds NATS-style wildcard pattern matching (*, **) for LiteTopic wildcard consumer groups, enabling selective subscription to a subset of lite-topics under a parent topic. It also optimizes the wildcard-group full dispatch from O(N) full LMQ table scan to O(M) parent-topic-scoped iteration. The implementation is well-structured with 18 new unit tests and preserves backward compatibility.

Findings

• [Warning] LitePatternMatcher.java — matches() calls validate() on every invocation. In expand(), this means the same pattern is re-validated for every candidate lite-topic. Consider splitting into an internal matchesUnchecked() and calling validate() once at the call site (e.g., in expand() or at pattern registration time). The PR description claims ~1µs/call, but eliminating redundant validation would improve throughput in hot dispatch paths.

• [Warning] LiteSubscriptionRegistryImpl.java:151 (aligns with Copilot) — When a client transitions from pattern-mode back to legacy mode (empty patterns) or to a non-wildcard group, the stale wildcardPatterns field on LiteSubscription is not cleared. This can cause doFullDispatchForWildcardGroup to misclassify the client as pattern-mode and reexpandWildcardPatterns to re-register stale LMQs. Suggested fix: explicitly call setWildcardPatterns(Collections.emptySet()) in the non-pattern branch of addCompleteSubscription.

• [Warning] LiteSubscriptionRegistryImpl.java:401-413 — doFullDispatchForWildcardGroup first calls getAllClientIdByGroup(group) then iterates to check each client's subscription. Between these two calls, a subscription could be removed or changed. Consider snapshotting the client list and subscription state under a single lock scope, or documenting that the broker event loop guarantees single-threaded access here.

• [Info] LitePatternMatcher.java:120-137 — expand() splits the pattern into segments via pattern.split(SEPARATOR, -1) on every call. For patterns reused across many candidates, pre-splitting once and passing the segments array would avoid repeated allocations.

• [Info] LiteSubscriptionRegistryImpl.java:430-457 — reexpandWildcardPatterns acquires and releases the read lock once per client. If many pattern-mode clients share the same parent topic, consider batching the expansion under a single lock acquisition to reduce lock contention.

• [Info] LiteSubscriptionCtlProcessor.java:107 (aligns with Copilot) — The lmqNameSet derived from pattern strings via LiteUtil.toLmqName() is semantically misleading since these are patterns, not actual lite-topic names. The set is passed to the registry but appears unused for pattern-mode groups. Consider passing Collections.emptySet() or null for the lmqNameAll parameter when in pattern mode to make the intent clearer.

Suggestions

1. Add a matchesUnchecked(String[] patternSegments, String liteTopic) package-private variant to avoid redundant validate() + split() in tight loops.
2. Clear wildcardPatterns on mode transition to prevent stale state bugs.
3. The cross-repo impact is minimal since patterns ride the existing liteTopicSet wire format — no changes needed in rocketmq-clients. 👍

Overall

Solid implementation with good test coverage and clean backward compatibility. The main concern is the stale wildcardPatterns on mode transition (shared with Copilot's finding). The repeated validate() calls are a performance optimization opportunity.

---

Automated review by github-manager-bot

[RockteMQ-AI]

Review by github-manager-bot (Re-review)

Summary

Three new commits since the initial review address dispatch gaps for newly-created lite-topics, add subscription quota enforcement, and fix quota projection ordering. The implementation is solid with comprehensive test coverage.

Changes Since Last Review

✅ Dispatch gap for new lite-topics — FIXED (a26ffbb5): Added registerArrivingLmqForPatternClients() which lazily registers an arriving lmqName against pattern-mode wildcard clients whose patterns match it. This closes the race window between lite-topic creation and the periodic doFullDispatchForWildcardGroup backstop. The addTopicGroup now returns boolean to track actual new entries — clean design.

✅ Subscription quota enforcement — ADDED (0e92110e + 2c597641): checkQuotaOrThrow() pre-flight check is now applied to all registration paths (non-wildcard complete-add, wildcard pattern expansion, and registerArrivingLmqForPatternClients). The quota projection was correctly moved before mutation in commit 2c597641 to avoid partial state on rejection. Good defensive programming.

✅ Mixed pattern/legacy wildcard groups — HANDLED: getAllSubscriber() now merges synthetic-key clients for legacy wildcard groups with pattern-mode clients. doFullDispatchForWildcardGroup partitions clients into pattern-mode and legacy subsets, dispatching each appropriately.

✅ Tests — EXCELLENT: New tests cover quota rejection on all paths, mixed-group delivery, deferred state cleanup on quota failure, and the lazy registration path. 138+ tests all green.

Addressed from Previous Review

• [Warning] expand() pre-splitting — ✅ Now uses compilePattern() to pre-split once and reuses the segments array across all candidates.
• [Warning] Stale wildcardPatterns — ✅ The deferredStateCleanup() helper now explicitly clears wildcardPatterns when transitioning out of pattern mode.
• [Info] Lock contention in reexpandWildcardPatterns — Partially addressed: the per-client expansion is now more targeted (only matching patterns), reducing the work under each lock acquisition.

Remaining Items (Non-blocking)

• [Warning] matches() still calls validate() on every invocation. While expand() now pre-splits, direct matches() callers (e.g., in registerArrivingLmqForPatternClients) still pay the validation cost per call. Consider caching validated patterns at registration time.

• [Info] The # separator used in LiteQuotaException messages is fine for logging but should not be used as a key delimiter in any future data structure (topics can contain # in some configurations).

Verdict

The three new commits significantly strengthen the implementation. Dispatch gaps are closed, quota is enforced consistently, and the code is well-tested. LGTM — ready for committer review.

---

Automated re-review by github-manager-bot