feat: opt-in CJK punctuation handling (./cjk + styles/cjk.css) - #90
Merged
Conversation
Full-width / ideographic punctuation (「」。！（）) is Unicode punctuation, so CommonMark's emphasis flanking rules never pair a `**` that sits between a CJK character and one of those marks — `これは**「強調」**です` stays literal, a shape LLMs emit constantly. This is a documented CommonMark limitation (the reference renderer produces the same output), not a bug here, so the fix ships as an opt-in extension (markdown-cjk-friendly), never the default. A post-process inline pass cannot fix it — the `**` are already left as text by the time passes run — so the extension is a default-off hook in the flanking classifier (`setFlankingPunctuationExclusion`) plus a matching bare-autolink boundary hook (`setBareUrlCjkBoundary`) that stops a run-together `https://example.com。次` at the first full-width mark. The new `@copse/streaming-markdown/cjk` entry (`setCjkFriendly`) flips both; its range table lives behind that entry, like the other optional backends, so it is never pulled into the default bundle. With it off, Latin output and the CommonMark / GFM conformance suites are byte-identical. Line-break and inter-script spacing are the host's CSS to own, not the renderer's: ship the optional `styles/cjk.css` (word-break / line-break / text-autospace, gated on `:lang()` or a `.sm-cjk` hook) and document the split in EXTENDING.md + README. Tests cover the classifier, emphasis/link adjacency, mixed CJK+Latin, and byte-identical Latin output; teardown resets the registry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ehmok4AzyTQDNkRzW3g8z8
# Conflicts: # docs/EXTENDING.md # package.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #85.
Opt-in CJK support, off by default (Latin output and both conformance suites byte-identical), gated like the other optional backends behind
@copse/streaming-markdown/cjk(verified absent from the core bundle).Scoping: what belongs in the renderer vs CSS
Studied
@streamdown/cjkand probed this renderer. Two genuine renderer gaps — both CommonMark extensions, not conformance bugs (the reference renderer produces identical output), hence opt-in:**「強調」**,**強調。**ですstay literal because CommonMark's flanking rule counts「」。！as blocking punctuation. Most CJK emphasis already works (ideographs are letters); only delimiters whose inner edge is CJK punctuation fail. A post-process pass can't fix this (the**are already left as text before passes run), so it needs a default-off hook in the flanking classifier that treats full-width punctuation as non-flanking.https://example.com。次absorbed。次into the href; now stops at the first full-width mark when enabled.Everything else is CSS the host owns — line breaking, Kinsoku, inter-script spacing, full-width kerning. The renderer emits identical structural HTML for every script and never guesses a language, so that ships as an optional
styles/cjk.css(word-break/line-break: strict/ progressivetext-autospace/text-spacing-trim, gated on:lang()or a.sm-cjkhook), not JS.API
setCjkFriendly()(opt-in) wires the two default-off hooks;isCjkPunctuation()exported.inline-emphasis.ts/inline-spans.tsgain default-nullinjectable hooks only; the behavior lives entirely in the./cjkentry.Verification
npm run typecheck,npm run build(dist/cjk.*emitted, smoke-tested) — passnpm test— 782 pass in isolation (CJK suite 13/13)npm run check:conformance/npm run check:gfm-conformance— 2/2, baselines byte-identicalnpm run coverage:ci— gate passes;cjk.ts100%; baseline unchanged at 99.75Caveats
nullhooks; justified in-commit because emphasis re-pairing structurally can't be a post-process pass.text-autospace,text-spacing-trim) that degrade safely where unsupported.🤖 Generated with Claude Code
https://claude.ai/code/session_01Ehmok4AzyTQDNkRzW3g8z8
Generated by Claude Code